[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:44:47 dist-test-slave-1nl6 krb5kdc[28706](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:44:47 dist-test-slave-1nl6 krb5kdc[28706](info): set up 2 sockets
Aug 10 00:44:47 dist-test-slave-1nl6 krb5kdc[28706](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260810 00:44:48.615600 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.300s user 0.002s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:44:48 dist-test-slave-1nl6 krb5kdc[28706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:44:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:44:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.6.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
I20260810 00:44:48.787623 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:36543
--webserver_interface=127.28.6.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:43755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:36543
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:44:49.118296 28720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:44:49.118566 28720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:44:49.118723 28720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:44:49.125842 28720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:44:49.125941 28720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:44:49.126046 28720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:44:49.126147 28720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:44:49.126245 28720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:44:49.135963 28720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:43755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:36543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:36543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:44:49.137404 28720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:44:49.138741 28720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:44:49.150995 28731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:44:49.151046 28732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:44:49.151188 28734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:44:49.151762 28720 server_base.cc:1034] running on GCE node
I20260810 00:44:49.152472 28720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:44:49.153812 28720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:44:49.154995 28720 hybrid_clock.cc:648] HybridClock initialized: now 1786322689154955 us; error 35 us; skew 500 ppm
I20260810 00:44:49.155297 28720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:44:49 dist-test-slave-1nl6 krb5kdc[28706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322689, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:44:49.158342 28720 init.cc:377] Logged in from keytab as kudu/127.28.6.62@KRBTEST.COM (short username kudu)
I20260810 00:44:49.159825 28720 webserver.cc:460] Webserver started at http://127.28.6.62:41651/ using document root <none> and password file <none>
I20260810 00:44:49.160508 28720 fs_manager.cc:359] Metadata directory not provided
I20260810 00:44:49.160640 28720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:44:49.160951 28720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:44:49.164687 28720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "d554c72a7c8a4d88ab00bb385c905043"
format_stamp: "Formatted at 2026-08-10 00:44:49 on dist-test-slave-1nl6"
server_key: "2f72b107a18b4aef102cdfae6813e629"
server_key_iv: "d456714c3135c046469f558d6916c859"
server_key_version: "encryptionkey@0"
I20260810 00:44:49.165465 28720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "d554c72a7c8a4d88ab00bb385c905043"
format_stamp: "Formatted at 2026-08-10 00:44:49 on dist-test-slave-1nl6"
server_key: "2f72b107a18b4aef102cdfae6813e629"
server_key_iv: "d456714c3135c046469f558d6916c859"
server_key_version: "encryptionkey@0"
I20260810 00:44:49.170466 28720 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 00:44:49.173856 28745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:44:49.175227 28720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 00:44:49.175417 28720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "d554c72a7c8a4d88ab00bb385c905043"
format_stamp: "Formatted at 2026-08-10 00:44:49 on dist-test-slave-1nl6"
server_key: "2f72b107a18b4aef102cdfae6813e629"
server_key_iv: "d456714c3135c046469f558d6916c859"
server_key_version: "encryptionkey@0"
I20260810 00:44:49.175580 28720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:44:49.202322 28720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:44:49.206029 28720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:44:49.206312 28720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:44:49.225215 28720 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:36543
I20260810 00:44:49.225263 28823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:36543 every 8 connection(s)
I20260810 00:44:49.226577 28720 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:44:49.230342 28824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:44:49.234760 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 28720
I20260810 00:44:49.235105 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:44:49.237553 28696 external_mini_cluster.cc:1428] Setting key 05589b2d8ba160c53a06f5844239cc03
I20260810 00:44:49.243180 28824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d554c72a7c8a4d88ab00bb385c905043: Bootstrap starting.
I20260810 00:44:49.246088 28824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d554c72a7c8a4d88ab00bb385c905043: Neither blocks nor log segments found. Creating new log.
I20260810 00:44:49.247159 28824 log.cc:824] T 00000000000000000000000000000000 P d554c72a7c8a4d88ab00bb385c905043: Log is configured to *not* fsync() on all Append() calls
I20260810 00:44:49.249745 28824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d554c72a7c8a4d88ab00bb385c905043: No bootstrap required, opened a new log
I20260810 00:44:49.259069 28824 raft_consensus.cc:348] T 00000000000000000000000000000000 P d554c72a7c8a4d88ab00bb385c905043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d554c72a7c8a4d88ab00bb385c905043" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36543 } }
I20260810 00:44:49.259521 28824 raft_consensus.cc:374] T 00000000000000000000000000000000 P d554c72a7c8a4d88ab00bb385c905043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:44:49.259626 28824 raft_consensus.cc:729] T 00000000000000000000000000000000 P d554c72a7c8a4d88ab00bb385c905043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d554c72a7c8a4d88ab00bb385c905043, State: Initialized, Role: FOLLOWER
I20260810 00:44:49.260267 28824 consensus_queue.cc:260] T 00000000000000000000000000000000 P d554c72a7c8a4d88ab00bb385c905043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d554c72a7c8a4d88ab00bb385c905043" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36543 } }
I20260810 00:44:49.260505 28824 raft_consensus.cc:388] T 00000000000000000000000000000000 P d554c72a7c8a4d88ab00bb385c905043 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:44:49.260653 28824 raft_consensus.cc:482] T 00000000000000000000000000000000 P d554c72a7c8a4d88ab00bb385c905043 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:44:49.260859 28824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d554c72a7c8a4d88ab00bb385c905043 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:44:49.263705 28824 raft_consensus.cc:504] T 00000000000000000000000000000000 P d554c72a7c8a4d88ab00bb385c905043 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d554c72a7c8a4d88ab00bb385c905043" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36543 } }
I20260810 00:44:49.264199 28824 leader_election.cc:302] T 00000000000000000000000000000000 P d554c72a7c8a4d88ab00bb385c905043 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d554c72a7c8a4d88ab00bb385c905043; no voters:
I20260810 00:44:49.264847 28824 leader_election.cc:288] T 00000000000000000000000000000000 P d554c72a7c8a4d88ab00bb385c905043 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:44:49.264986 28832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d554c72a7c8a4d88ab00bb385c905043 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:44:49.266237 28832 raft_consensus.cc:686] T 00000000000000000000000000000000 P d554c72a7c8a4d88ab00bb385c905043 [term 1 LEADER]: Becoming Leader. State: Replica: d554c72a7c8a4d88ab00bb385c905043, State: Running, Role: LEADER
I20260810 00:44:49.266973 28824 sys_catalog.cc:564] T 00000000000000000000000000000000 P d554c72a7c8a4d88ab00bb385c905043 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:44:49.266878 28832 consensus_queue.cc:237] T 00000000000000000000000000000000 P d554c72a7c8a4d88ab00bb385c905043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d554c72a7c8a4d88ab00bb385c905043" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36543 } }
Aug 10 00:44:49 dist-test-slave-1nl6 krb5kdc[28706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:44:49.270097 28834 sys_catalog.cc:455] T 00000000000000000000000000000000 P d554c72a7c8a4d88ab00bb385c905043 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d554c72a7c8a4d88ab00bb385c905043. Latest consensus state: current_term: 1 leader_uuid: "d554c72a7c8a4d88ab00bb385c905043" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d554c72a7c8a4d88ab00bb385c905043" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36543 } } }
I20260810 00:44:49.270563 28834 sys_catalog.cc:458] T 00000000000000000000000000000000 P d554c72a7c8a4d88ab00bb385c905043 [sys.catalog]: This master's current role is: LEADER
I20260810 00:44:49.271142 28833 sys_catalog.cc:455] T 00000000000000000000000000000000 P d554c72a7c8a4d88ab00bb385c905043 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d554c72a7c8a4d88ab00bb385c905043" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d554c72a7c8a4d88ab00bb385c905043" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36543 } } }
I20260810 00:44:49.271857 28833 sys_catalog.cc:458] T 00000000000000000000000000000000 P d554c72a7c8a4d88ab00bb385c905043 [sys.catalog]: This master's current role is: LEADER
I20260810 00:44:49.285007 28827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:49.242146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35118 (local address 127.28.6.62:36543)
0810 00:44:49.243094 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0810 00:44:49.243109 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 00:44:49.244526 (+ 1417us) server_negotiation.cc:407] Connection header received
0810 00:44:49.254468 (+ 9942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:44:49.254495 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:44:49.254975 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:44:49.255413 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:44:49.258077 (+ 2664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:49.260080 (+ 2003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:49.261917 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:49.263037 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:49.263221 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:49.270636 (+ 7415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:44:49.270650 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:44:49.270669 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:44:49.270720 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:44:49.275802 (+ 5082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:49.277094 (+ 1292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:49.277106 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:49.277118 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:49.277393 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:49.278108 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:49.278120 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:49.278127 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:49.279106 (+ 979us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:44:49.279487 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0810 00:44:49.282723 (+ 3236us) server_negotiation.cc:299] Negotiation successful
0810 00:44:49.283308 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":436,"threads_started":1}
I20260810 00:44:49.288504 28845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:44:49.293607 28845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:44:49.301779 28845 catalog_manager.cc:1347] Generated new cluster ID: b9a3a637da5f440eb3cbed5391b17b3e
I20260810 00:44:49.301891 28845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:44:49.312608 28845 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:44:49.314407 28845 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:44:49.323108 28845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d554c72a7c8a4d88ab00bb385c905043: Generated new TSK 0
I20260810 00:44:49.324015 28845 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.6.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab.
Entry for principal kudu/127.28.6.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab.
I20260810 00:44:49.368110 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.1:0
--local_ip_for_outbound_sockets=127.28.6.1
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:36543
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:43755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:44:49.921572 28860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:44:49.921902 28860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:44:49.922119 28860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:44:49.931141 28860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:44:49.931288 28860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:44:49.931452 28860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.1
I20260810 00:44:49.942970 28860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:43755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:36543
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.1
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:44:49.944321 28860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:44:49.945796 28860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:44:49.963006 28869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:44:49.963810 28874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:44:49.964043 28868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:44:49.964533 28860 server_base.cc:1034] running on GCE node
I20260810 00:44:49.965163 28860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:44:49.966113 28860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:44:49.967329 28860 hybrid_clock.cc:648] HybridClock initialized: now 1786322689967247 us; error 40 us; skew 500 ppm
I20260810 00:44:49.967646 28860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:44:49 dist-test-slave-1nl6 krb5kdc[28706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322689, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:44:49.971503 28860 init.cc:377] Logged in from keytab as kudu/127.28.6.1@KRBTEST.COM (short username kudu)
I20260810 00:44:49.973338 28860 webserver.cc:460] Webserver started at http://127.28.6.1:46329/ using document root <none> and password file <none>
I20260810 00:44:49.974057 28860 fs_manager.cc:359] Metadata directory not provided
I20260810 00:44:49.974268 28860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:44:49.974659 28860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:44:49.978186 28860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-0/data/instance:
uuid: "eda4fe4ea8924935be4d403aa0f40f91"
format_stamp: "Formatted at 2026-08-10 00:44:49 on dist-test-slave-1nl6"
server_key: "91c06dad976b7a8376fc12b111aa095e"
server_key_iv: "4703993954548513eb7170e01096025f"
server_key_version: "encryptionkey@0"
I20260810 00:44:49.979182 28860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance:
uuid: "eda4fe4ea8924935be4d403aa0f40f91"
format_stamp: "Formatted at 2026-08-10 00:44:49 on dist-test-slave-1nl6"
server_key: "91c06dad976b7a8376fc12b111aa095e"
server_key_iv: "4703993954548513eb7170e01096025f"
server_key_version: "encryptionkey@0"
I20260810 00:44:49.985603 28860 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260810 00:44:49.990047 28881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:44:49.991484 28860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 00:44:49.991691 28860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-0/wal
uuid: "eda4fe4ea8924935be4d403aa0f40f91"
format_stamp: "Formatted at 2026-08-10 00:44:49 on dist-test-slave-1nl6"
server_key: "91c06dad976b7a8376fc12b111aa095e"
server_key_iv: "4703993954548513eb7170e01096025f"
server_key_version: "encryptionkey@0"
I20260810 00:44:49.991961 28860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:44:50.015115 28860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:44:50.020049 28860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:44:50.020507 28860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:44:50.021824 28860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:44:50.023710 28860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:44:50.023869 28860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:44:50.024044 28860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:44:50.024163 28860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:44:50.096174 28860 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.1:42439
I20260810 00:44:50.096282 29058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.1:42439 every 8 connection(s)
I20260810 00:44:50.097633 28860 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
I20260810 00:44:50.099514 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 28860
I20260810 00:44:50.099800 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance
I20260810 00:44:50.100777 28696 external_mini_cluster.cc:1428] Setting key bbea4787bd4150a95cd6389b3b802374
Aug 10 00:44:50 dist-test-slave-1nl6 krb5kdc[28706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322689, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.6.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.2@KRBTEST.COM" created.
I20260810 00:44:50.125676 29061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:50.101256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:46891 (local address 127.28.6.62:36543)
0810 00:44:50.102055 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0810 00:44:50.102060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:44:50.103870 (+ 1810us) server_negotiation.cc:407] Connection header received
0810 00:44:50.106186 (+ 2316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:44:50.106194 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:44:50.106339 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:44:50.106464 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:44:50.108551 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:50.109489 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:50.111488 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:50.112011 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:50.112148 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:50.116905 (+ 4757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:44:50.116914 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:44:50.116921 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:44:50.117136 (+ 215us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:44:50.121773 (+ 4637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:50.122751 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:50.122760 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:50.122766 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:50.123033 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:50.123793 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:50.123799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:50.123802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:50.124122 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:44:50.124285 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 00:44:50.125042 (+ 757us) server_negotiation.cc:299] Negotiation successful
0810 00:44:50.125300 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":513,"threads_started":1}
I20260810 00:44:50.126973 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:50.102122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36543 (local address 127.28.6.1:46891)
0810 00:44:50.103674 (+ 1552us) negotiation.cc:107] Waiting for socket to connect
0810 00:44:50.103738 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 00:44:50.105894 (+ 2156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:44:50.106698 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:44:50.106706 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:44:50.107414 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:44:50.108311 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:44:50.108351 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:44:50.109686 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:44:50.109690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:44:50.111321 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:44:50.111335 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:44:50.112378 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:44:50.112390 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:44:50.112685 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:50.113478 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:44:50.113519 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:44:50.116566 (+ 3047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:44:50.121931 (+ 5365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:44:50.121938 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:44:50.121952 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:44:50.122549 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:44:50.123195 (+ 646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:44:50.123253 (+ 58us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:44:50.123261 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:44:50.123652 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:44:50.124267 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:44:50.124275 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:44:50.124640 (+ 365us) client_negotiation.cc:769] Sending connection context
0810 00:44:50.124926 (+ 286us) client_negotiation.cc:240] Negotiation successful
0810 00:44:50.125535 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1195,"thread_start_us":792,"threads_started":1}
I20260810 00:44:50.128720 29059 heartbeater.cc:347] Connected to a master server at 127.28.6.62:36543
I20260810 00:44:50.129136 29059 heartbeater.cc:464] Registering TS with master...
I20260810 00:44:50.130501 29059 heartbeater.cc:511] Master 127.28.6.62:36543 requested a full tablet report, sending...
I20260810 00:44:50.133333 28767 ts_manager.cc:194] Registered new tserver with Master: eda4fe4ea8924935be4d403aa0f40f91 (127.28.6.1:42439)
I20260810 00:44:50.136175 28767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.1@KRBTEST.COM'} at 127.28.6.1:46891
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab.
Entry for principal kudu/127.28.6.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab.
I20260810 00:44:50.150499 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.2:0
--local_ip_for_outbound_sockets=127.28.6.2
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:36543
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:43755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:44:50.666327 29069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:44:50.666687 29069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:44:50.666915 29069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:44:50.675917 29069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:44:50.676126 29069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:44:50.676352 29069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.2
I20260810 00:44:50.688923 29069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:43755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:36543
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.2
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:44:50.690266 29069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:44:50.691860 29069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:44:50.710052 29080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:44:50.711048 29077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:44:50.711315 29069 server_base.cc:1034] running on GCE node
W20260810 00:44:50.711575 29074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:44:50.712493 29069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:44:50.713903 29069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:44:50.715214 29069 hybrid_clock.cc:648] HybridClock initialized: now 1786322690715144 us; error 65 us; skew 500 ppm
I20260810 00:44:50.715663 29069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:44:50 dist-test-slave-1nl6 krb5kdc[28706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322690, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:44:50.721024 29069 init.cc:377] Logged in from keytab as kudu/127.28.6.2@KRBTEST.COM (short username kudu)
I20260810 00:44:50.723168 29069 webserver.cc:460] Webserver started at http://127.28.6.2:37875/ using document root <none> and password file <none>
I20260810 00:44:50.724009 29069 fs_manager.cc:359] Metadata directory not provided
I20260810 00:44:50.724243 29069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:44:50.724689 29069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:44:50.728402 29069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-1/data/instance:
uuid: "87d9b9cb6d49416586dbf33489d01ac4"
format_stamp: "Formatted at 2026-08-10 00:44:50 on dist-test-slave-1nl6"
server_key: "430b0e27e0aa67c89f2baa9e8869c003"
server_key_iv: "cc85e54c5b33edb0350219294da2f97a"
server_key_version: "encryptionkey@0"
I20260810 00:44:50.729269 29069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance:
uuid: "87d9b9cb6d49416586dbf33489d01ac4"
format_stamp: "Formatted at 2026-08-10 00:44:50 on dist-test-slave-1nl6"
server_key: "430b0e27e0aa67c89f2baa9e8869c003"
server_key_iv: "cc85e54c5b33edb0350219294da2f97a"
server_key_version: "encryptionkey@0"
I20260810 00:44:50.735448 29069 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.007s
I20260810 00:44:50.739853 29092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:44:50.741278 29069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 00:44:50.741621 29069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-1/wal
uuid: "87d9b9cb6d49416586dbf33489d01ac4"
format_stamp: "Formatted at 2026-08-10 00:44:50 on dist-test-slave-1nl6"
server_key: "430b0e27e0aa67c89f2baa9e8869c003"
server_key_iv: "cc85e54c5b33edb0350219294da2f97a"
server_key_version: "encryptionkey@0"
I20260810 00:44:50.742007 29069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:44:50.768839 29069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:44:50.773720 29069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:44:50.774179 29069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:44:50.775803 29069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:44:50.777487 29069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:44:50.777637 29069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:44:50.777828 29069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:44:50.777917 29069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:44:50.837927 29069 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.2:33669
I20260810 00:44:50.838027 29265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.2:33669 every 8 connection(s)
I20260810 00:44:50.839541 29069 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
I20260810 00:44:50.841200 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 29069
I20260810 00:44:50.841475 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance
I20260810 00:44:50.842211 28696 external_mini_cluster.cc:1428] Setting key 6921240dca804de2b50180b4a243ea29
Aug 10 00:44:50 dist-test-slave-1nl6 krb5kdc[28706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322690, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.6.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.3@KRBTEST.COM" created.
I20260810 00:44:50.873696 29272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:50.848015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:60979 (local address 127.28.6.62:36543)
0810 00:44:50.849235 (+ 1220us) server_negotiation.cc:206] Beginning negotiation
0810 00:44:50.849244 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:44:50.850144 (+ 900us) server_negotiation.cc:407] Connection header received
0810 00:44:50.853080 (+ 2936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:44:50.853084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:44:50.853244 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:44:50.853346 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:44:50.855893 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:50.856658 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:50.860189 (+ 3531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:50.860660 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:50.860778 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:50.865867 (+ 5089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:44:50.865871 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:44:50.865875 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:44:50.865906 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:44:50.870112 (+ 4206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:50.870875 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:50.870884 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:50.870890 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:50.871125 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:50.871629 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:50.871637 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:50.871642 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:50.872098 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:44:50.872275 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0810 00:44:50.873050 (+ 775us) server_negotiation.cc:299] Negotiation successful
0810 00:44:50.873287 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1058,"thread_start_us":1199,"threads_started":1}
I20260810 00:44:50.874768 29274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:50.848652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36543 (local address 127.28.6.2:60979)
0810 00:44:50.849958 (+ 1306us) negotiation.cc:107] Waiting for socket to connect
0810 00:44:50.850015 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 00:44:50.852750 (+ 2735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:44:50.853624 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:44:50.853638 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:44:50.854545 (+ 907us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:44:50.855713 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:44:50.855742 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:44:50.856871 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:44:50.856879 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:44:50.859651 (+ 2772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:44:50.859672 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:44:50.860909 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:44:50.860920 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:44:50.861394 (+ 474us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:50.862303 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:44:50.862343 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:44:50.865640 (+ 3297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:44:50.870249 (+ 4609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:44:50.870256 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:44:50.870267 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:44:50.870735 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:44:50.871248 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:44:50.871252 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:44:50.871254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:44:50.871500 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:44:50.872291 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:44:50.872297 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:44:50.872632 (+ 335us) client_negotiation.cc:769] Sending connection context
0810 00:44:50.872938 (+ 306us) client_negotiation.cc:240] Negotiation successful
0810 00:44:50.873534 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":976,"thread_start_us":674,"threads_started":1}
I20260810 00:44:50.876252 29267 heartbeater.cc:347] Connected to a master server at 127.28.6.62:36543
I20260810 00:44:50.876582 29267 heartbeater.cc:464] Registering TS with master...
I20260810 00:44:50.877394 29267 heartbeater.cc:511] Master 127.28.6.62:36543 requested a full tablet report, sending...
I20260810 00:44:50.879372 28767 ts_manager.cc:194] Registered new tserver with Master: 87d9b9cb6d49416586dbf33489d01ac4 (127.28.6.2:33669)
I20260810 00:44:50.880932 28767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.2@KRBTEST.COM'} at 127.28.6.2:60979
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab.
Entry for principal kudu/127.28.6.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab.
I20260810 00:44:50.891413 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.3:0
--local_ip_for_outbound_sockets=127.28.6.3
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:36543
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:43755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 00:44:51.140755 29059 heartbeater.cc:503] Master 127.28.6.62:36543 was elected leader, sending a full tablet report...
W20260810 00:44:51.425685 29279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:44:51.426077 29279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:44:51.426328 29279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:44:51.435813 29279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:44:51.436053 29279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:44:51.436306 29279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.3
I20260810 00:44:51.447980 29279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:43755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:36543
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.3
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:44:51.449316 29279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:44:51.450738 29279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:44:51.468678 29288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:44:51.469331 29293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:44:51.469525 29289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:44:51.471263 29279 server_base.cc:1034] running on GCE node
I20260810 00:44:51.472000 29279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:44:51.473114 29279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:44:51.474439 29279 hybrid_clock.cc:648] HybridClock initialized: now 1786322691474336 us; error 41 us; skew 500 ppm
I20260810 00:44:51.474759 29279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:44:51 dist-test-slave-1nl6 krb5kdc[28706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322691, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:44:51.478982 29279 init.cc:377] Logged in from keytab as kudu/127.28.6.3@KRBTEST.COM (short username kudu)
I20260810 00:44:51.481338 29279 webserver.cc:460] Webserver started at http://127.28.6.3:36495/ using document root <none> and password file <none>
I20260810 00:44:51.482149 29279 fs_manager.cc:359] Metadata directory not provided
I20260810 00:44:51.482363 29279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:44:51.482792 29279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:44:51.486416 29279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-2/data/instance:
uuid: "1720886b2aea466598edad830632d524"
format_stamp: "Formatted at 2026-08-10 00:44:51 on dist-test-slave-1nl6"
server_key: "d2d9ab1ec7295603e45b8252d9b365de"
server_key_iv: "45530f826636666949a385a4c1cc7390"
server_key_version: "encryptionkey@0"
I20260810 00:44:51.487421 29279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance:
uuid: "1720886b2aea466598edad830632d524"
format_stamp: "Formatted at 2026-08-10 00:44:51 on dist-test-slave-1nl6"
server_key: "d2d9ab1ec7295603e45b8252d9b365de"
server_key_iv: "45530f826636666949a385a4c1cc7390"
server_key_version: "encryptionkey@0"
I20260810 00:44:51.493732 29279 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260810 00:44:51.498287 29303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:44:51.499806 29279 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 00:44:51.500015 29279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-2/wal
uuid: "1720886b2aea466598edad830632d524"
format_stamp: "Formatted at 2026-08-10 00:44:51 on dist-test-slave-1nl6"
server_key: "d2d9ab1ec7295603e45b8252d9b365de"
server_key_iv: "45530f826636666949a385a4c1cc7390"
server_key_version: "encryptionkey@0"
I20260810 00:44:51.500305 29279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:44:51.526765 29279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:44:51.531669 29279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:44:51.532119 29279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:44:51.533476 29279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:44:51.535518 29279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:44:51.535662 29279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:44:51.535923 29279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:44:51.536070 29279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:44:51.605050 29279 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.3:41427
I20260810 00:44:51.605391 29480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.3:41427 every 8 connection(s)
I20260810 00:44:51.606542 29279 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
I20260810 00:44:51.612228 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 29279
I20260810 00:44:51.612586 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance
I20260810 00:44:51.613353 28696 external_mini_cluster.cc:1428] Setting key f8f38134ed037c29ce71a878f3994ff4
Aug 10 00:44:51 dist-test-slave-1nl6 krb5kdc[28706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322691, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:44:51.636899 29483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:51.614669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:49915 (local address 127.28.6.62:36543)
0810 00:44:51.615544 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0810 00:44:51.615550 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:44:51.617041 (+ 1491us) server_negotiation.cc:407] Connection header received
0810 00:44:51.619618 (+ 2577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:44:51.619624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:44:51.619795 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:44:51.619922 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:44:51.621859 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:51.622801 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:51.624760 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:51.625383 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:51.625524 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:51.630179 (+ 4655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:44:51.630185 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:44:51.630188 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:44:51.630219 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:44:51.633837 (+ 3618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:51.634592 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:51.634597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:51.634599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:51.634759 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:51.635243 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:51.635247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:51.635249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:51.635540 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:44:51.635651 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 00:44:51.636314 (+ 663us) server_negotiation.cc:299] Negotiation successful
0810 00:44:51.636490 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":537,"threads_started":1}
I20260810 00:44:51.637894 29484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:51.615454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36543 (local address 127.28.6.3:49915)
0810 00:44:51.616892 (+ 1438us) negotiation.cc:107] Waiting for socket to connect
0810 00:44:51.616939 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 00:44:51.619355 (+ 2416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:44:51.620249 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:44:51.620257 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:44:51.620920 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:44:51.621694 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:44:51.621713 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:44:51.622981 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:44:51.622986 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:44:51.624607 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:44:51.624619 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:44:51.625707 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:44:51.625720 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:44:51.625997 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:51.626839 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:44:51.626884 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:44:51.629943 (+ 3059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:44:51.633966 (+ 4023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:44:51.633973 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:44:51.633984 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:44:51.634471 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:44:51.634859 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:44:51.634862 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:44:51.634864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:44:51.635122 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:44:51.635643 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:44:51.635649 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:44:51.635985 (+ 336us) client_negotiation.cc:769] Sending connection context
0810 00:44:51.636220 (+ 235us) client_negotiation.cc:240] Negotiation successful
0810 00:44:51.636629 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":995,"thread_start_us":645,"threads_started":1}
I20260810 00:44:51.639384 29481 heartbeater.cc:347] Connected to a master server at 127.28.6.62:36543
I20260810 00:44:51.639751 29481 heartbeater.cc:464] Registering TS with master...
I20260810 00:44:51.640635 29481 heartbeater.cc:511] Master 127.28.6.62:36543 requested a full tablet report, sending...
I20260810 00:44:51.642544 28767 ts_manager.cc:194] Registered new tserver with Master: 1720886b2aea466598edad830632d524 (127.28.6.3:41427)
I20260810 00:44:51.643332 28696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 00:44:51.643579 28767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.3@KRBTEST.COM'} at 127.28.6.3:49915
Aug 10 00:44:51 dist-test-slave-1nl6 krb5kdc[28706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322691, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 00:44:51 dist-test-slave-1nl6 krb5kdc[28706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322691, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.6.1@KRBTEST.COM
I20260810 00:44:51.696609 29493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:51.677654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58010 (local address 127.28.6.1:42439)
0810 00:44:51.678644 (+ 990us) server_negotiation.cc:206] Beginning negotiation
0810 00:44:51.678657 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 00:44:51.679103 (+ 446us) server_negotiation.cc:407] Connection header received
0810 00:44:51.679346 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:44:51.679354 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:44:51.679592 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:44:51.679753 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:44:51.681499 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:51.682493 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:51.684054 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:51.684586 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:51.684719 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:51.689936 (+ 5217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:44:51.689948 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:44:51.689962 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:44:51.690001 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:44:51.693543 (+ 3542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:51.694313 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:51.694320 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:51.694323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:51.694497 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:51.695020 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:51.695024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:51.695025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:51.695412 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:44:51.695719 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0810 00:44:51.695853 (+ 134us) server_negotiation.cc:299] Negotiation successful
0810 00:44:51.696096 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":387,"threads_started":1}
W20260810 00:44:51.698122 28919 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58010
Aug 10 00:44:51 dist-test-slave-1nl6 krb5kdc[28706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 00:44:51 dist-test-slave-1nl6 krb5kdc[28706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.1@KRBTEST.COM
I20260810 00:44:51.749372 29493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:51.733427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58020 (local address 127.28.6.1:42439)
0810 00:44:51.733620 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 00:44:51.733625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:44:51.733645 (+ 20us) server_negotiation.cc:407] Connection header received
0810 00:44:51.733786 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:44:51.733790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:44:51.733949 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:44:51.734052 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:44:51.735723 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:51.736568 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:51.737919 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:51.738479 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:51.738576 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:51.743379 (+ 4803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:44:51.743384 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:44:51.743387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:44:51.743420 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:44:51.747012 (+ 3592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:51.747716 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:51.747722 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:51.747725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:51.747861 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:51.748217 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:51.748220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:51.748222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:51.748452 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:44:51.748622 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0810 00:44:51.748920 (+ 298us) server_negotiation.cc:299] Negotiation successful
0810 00:44:51.749076 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 00:44:51.751221 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 28860
I20260810 00:44:51.765901 29493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:51.752262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58028 (local address 127.28.6.1:42439)
0810 00:44:51.752461 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 00:44:51.752468 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:44:51.752676 (+ 208us) server_negotiation.cc:407] Connection header received
0810 00:44:51.752851 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:44:51.752861 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:44:51.752914 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:44:51.753008 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:44:51.754148 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:51.754848 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:51.756280 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:51.756836 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:51.756972 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:51.759958 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:44:51.759964 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:44:51.759967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:44:51.759996 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:44:51.763158 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:51.763767 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:51.763777 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:51.763779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:51.763984 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:51.764347 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:51.764352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:51.764355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:51.764650 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:44:51.764949 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0810 00:44:51.765218 (+ 269us) server_negotiation.cc:299] Negotiation successful
0810 00:44:51.765443 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 00:44:51.766746 29052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:44:51.853499 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 28860
I20260810 00:44:51.881577 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 29069
I20260810 00:44:51.883932 29267 heartbeater.cc:503] Master 127.28.6.62:36543 was elected leader, sending a full tablet report...
Aug 10 00:44:51 dist-test-slave-1nl6 krb5kdc[28706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.2@KRBTEST.COM
I20260810 00:44:51.898365 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:51.882593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59788 (local address 127.28.6.2:33669)
0810 00:44:51.883228 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0810 00:44:51.883236 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:44:51.883261 (+ 25us) server_negotiation.cc:407] Connection header received
0810 00:44:51.883323 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:44:51.883329 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:44:51.883528 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:44:51.883692 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:44:51.885411 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:51.886577 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:51.887908 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:51.888619 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:51.888719 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:51.893262 (+ 4543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:44:51.893273 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:44:51.893287 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:44:51.893321 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:44:51.896028 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:51.896594 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:51.896601 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:51.896603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:51.896722 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:51.897058 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:51.897061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:51.897063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:51.897381 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:44:51.897644 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0810 00:44:51.897889 (+ 245us) server_negotiation.cc:299] Negotiation successful
0810 00:44:51.898078 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"spinlock_wait_cycles":27392,"thread_start_us":348,"threads_started":1}
I20260810 00:44:51.899842 29257 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:44:51.982635 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 29069
I20260810 00:44:52.009303 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 29279
Aug 10 00:44:52 dist-test-slave-1nl6 krb5kdc[28706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.3@KRBTEST.COM
I20260810 00:44:52.025729 29508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:52.010389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52860 (local address 127.28.6.3:41427)
0810 00:44:52.010955 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0810 00:44:52.010962 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:44:52.010993 (+ 31us) server_negotiation.cc:407] Connection header received
0810 00:44:52.011061 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:44:52.011067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:44:52.011293 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:44:52.011421 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:44:52.012609 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:52.013616 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:52.014879 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:52.015609 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:52.015712 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:52.020498 (+ 4786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:44:52.020508 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:44:52.020522 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:44:52.020553 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:44:52.023254 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:52.023885 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:52.023892 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:52.023894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:52.024015 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:52.024327 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:52.024331 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:52.024332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:52.024665 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:44:52.024881 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0810 00:44:52.025134 (+ 253us) server_negotiation.cc:299] Negotiation successful
0810 00:44:52.025334 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":250,"threads_started":1}
I20260810 00:44:52.027041 29476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:44:52.109422 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 29279
I20260810 00:44:52.132344 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 28720
I20260810 00:44:52.133565 28817 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:44:52.236554 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 28720
2026-08-10T00:44:52Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4979 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:44:52 dist-test-slave-1nl6 krb5kdc[29514](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:44:52 dist-test-slave-1nl6 krb5kdc[29514](info): set up 2 sockets
Aug 10 00:44:52 dist-test-slave-1nl6 krb5kdc[29514](info): commencing operation
krb5kdc: starting...
W20260810 00:44:53.522378 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.238s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:44:53 dist-test-slave-1nl6 krb5kdc[29514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:44:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:44:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.6.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
I20260810 00:44:53.704267 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:36471
--webserver_interface=127.28.6.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:46231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:36471
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:44:54.025346 29535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:44:54.025625 29535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:44:54.025800 29535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:44:54.033074 29535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:44:54.033177 29535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:44:54.033298 29535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:44:54.033389 29535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:44:54.033468 29535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:44:54.043272 29535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:46231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:36471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:36471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:44:54.044595 29535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:44:54.045918 29535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:44:54.057549 29541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:44:54.057647 29542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:44:54.057612 29546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:44:54.060169 29535 server_base.cc:1034] running on GCE node
I20260810 00:44:54.060997 29535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:44:54.062347 29535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:44:54.063599 29535 hybrid_clock.cc:648] HybridClock initialized: now 1786322694063553 us; error 46 us; skew 500 ppm
I20260810 00:44:54.063961 29535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:44:54 dist-test-slave-1nl6 krb5kdc[29514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322694, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:44:54.071530 29535 init.cc:377] Logged in from keytab as kudu/127.28.6.62@KRBTEST.COM (short username kudu)
I20260810 00:44:54.073179 29535 webserver.cc:460] Webserver started at http://127.28.6.62:38587/ using document root <none> and password file <none>
I20260810 00:44:54.073931 29535 fs_manager.cc:359] Metadata directory not provided
I20260810 00:44:54.074090 29535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:44:54.074425 29535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:44:54.077564 29535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "23488c096c6f44b1b175f1a35e5e2dc1"
format_stamp: "Formatted at 2026-08-10 00:44:54 on dist-test-slave-1nl6"
server_key: "4103c799d20d60522182606a03434746"
server_key_iv: "4e1d73869c23212963f050c7f1b0e9ed"
server_key_version: "encryptionkey@0"
I20260810 00:44:54.078356 29535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "23488c096c6f44b1b175f1a35e5e2dc1"
format_stamp: "Formatted at 2026-08-10 00:44:54 on dist-test-slave-1nl6"
server_key: "4103c799d20d60522182606a03434746"
server_key_iv: "4e1d73869c23212963f050c7f1b0e9ed"
server_key_version: "encryptionkey@0"
I20260810 00:44:54.083798 29535 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 00:44:54.087487 29556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:44:54.088896 29535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 00:44:54.089099 29535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "23488c096c6f44b1b175f1a35e5e2dc1"
format_stamp: "Formatted at 2026-08-10 00:44:54 on dist-test-slave-1nl6"
server_key: "4103c799d20d60522182606a03434746"
server_key_iv: "4e1d73869c23212963f050c7f1b0e9ed"
server_key_version: "encryptionkey@0"
I20260810 00:44:54.089315 29535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:44:54.102856 29535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:44:54.107317 29535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:44:54.107611 29535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:44:54.126089 29535 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:36471
I20260810 00:44:54.126124 29633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:36471 every 8 connection(s)
I20260810 00:44:54.127475 29535 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:44:54.131107 29634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:44:54.131765 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 29535
I20260810 00:44:54.132004 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:44:54.132654 28696 external_mini_cluster.cc:1428] Setting key 6b29edb3f8274a780ba84a4029696d6c
I20260810 00:44:54.147329 29634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23488c096c6f44b1b175f1a35e5e2dc1: Bootstrap starting.
Aug 10 00:44:54 dist-test-slave-1nl6 krb5kdc[29514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:44:54.151202 29634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23488c096c6f44b1b175f1a35e5e2dc1: Neither blocks nor log segments found. Creating new log.
I20260810 00:44:54.152274 29634 log.cc:824] T 00000000000000000000000000000000 P 23488c096c6f44b1b175f1a35e5e2dc1: Log is configured to *not* fsync() on all Append() calls
I20260810 00:44:54.155395 29634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23488c096c6f44b1b175f1a35e5e2dc1: No bootstrap required, opened a new log
I20260810 00:44:54.158883 29638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:54.134700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37730 (local address 127.28.6.62:36471)
0810 00:44:54.135690 (+ 990us) server_negotiation.cc:206] Beginning negotiation
0810 00:44:54.135706 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 00:44:54.135756 (+ 50us) server_negotiation.cc:407] Connection header received
0810 00:44:54.138196 (+ 2440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:44:54.138227 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:44:54.138740 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:44:54.139243 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:44:54.141243 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:54.143369 (+ 2126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:54.145352 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:54.146422 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:54.146606 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:54.151372 (+ 4766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:44:54.151384 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:44:54.151401 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:44:54.151449 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:44:54.154851 (+ 3402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:54.155551 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:54.155559 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:54.155567 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:54.155741 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:54.156147 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:54.156152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:54.156156 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:54.156656 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:44:54.157018 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0810 00:44:54.157490 (+ 472us) server_negotiation.cc:299] Negotiation successful
0810 00:44:54.157897 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":740,"thread_start_us":442,"threads_started":1}
I20260810 00:44:54.163872 29634 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23488c096c6f44b1b175f1a35e5e2dc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23488c096c6f44b1b175f1a35e5e2dc1" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36471 } }
I20260810 00:44:54.164240 29634 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23488c096c6f44b1b175f1a35e5e2dc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:44:54.164357 29634 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23488c096c6f44b1b175f1a35e5e2dc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23488c096c6f44b1b175f1a35e5e2dc1, State: Initialized, Role: FOLLOWER
I20260810 00:44:54.164973 29634 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23488c096c6f44b1b175f1a35e5e2dc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23488c096c6f44b1b175f1a35e5e2dc1" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36471 } }
I20260810 00:44:54.165274 29634 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23488c096c6f44b1b175f1a35e5e2dc1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:44:54.165388 29634 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23488c096c6f44b1b175f1a35e5e2dc1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:44:54.165506 29634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23488c096c6f44b1b175f1a35e5e2dc1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:44:54.167541 29634 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23488c096c6f44b1b175f1a35e5e2dc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23488c096c6f44b1b175f1a35e5e2dc1" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36471 } }
I20260810 00:44:54.167971 29634 leader_election.cc:302] T 00000000000000000000000000000000 P 23488c096c6f44b1b175f1a35e5e2dc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23488c096c6f44b1b175f1a35e5e2dc1; no voters:
I20260810 00:44:54.168510 29634 leader_election.cc:288] T 00000000000000000000000000000000 P 23488c096c6f44b1b175f1a35e5e2dc1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:44:54.168605 29641 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23488c096c6f44b1b175f1a35e5e2dc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:44:54.169313 29641 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23488c096c6f44b1b175f1a35e5e2dc1 [term 1 LEADER]: Becoming Leader. State: Replica: 23488c096c6f44b1b175f1a35e5e2dc1, State: Running, Role: LEADER
I20260810 00:44:54.169983 29641 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23488c096c6f44b1b175f1a35e5e2dc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23488c096c6f44b1b175f1a35e5e2dc1" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36471 } }
I20260810 00:44:54.170692 29634 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23488c096c6f44b1b175f1a35e5e2dc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:44:54.173132 29647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23488c096c6f44b1b175f1a35e5e2dc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23488c096c6f44b1b175f1a35e5e2dc1. Latest consensus state: current_term: 1 leader_uuid: "23488c096c6f44b1b175f1a35e5e2dc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23488c096c6f44b1b175f1a35e5e2dc1" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36471 } } }
I20260810 00:44:54.173655 29647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23488c096c6f44b1b175f1a35e5e2dc1 [sys.catalog]: This master's current role is: LEADER
I20260810 00:44:54.173157 29646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23488c096c6f44b1b175f1a35e5e2dc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23488c096c6f44b1b175f1a35e5e2dc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23488c096c6f44b1b175f1a35e5e2dc1" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36471 } } }
I20260810 00:44:54.174587 29646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23488c096c6f44b1b175f1a35e5e2dc1 [sys.catalog]: This master's current role is: LEADER
I20260810 00:44:54.176034 29654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:44:54.181766 29654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:44:54.190424 29654 catalog_manager.cc:1347] Generated new cluster ID: 6b66790390fe4d079f9857994db8196f
I20260810 00:44:54.190534 29654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:44:54.217880 29654 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:44:54.219699 29654 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:44:54.228073 29654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23488c096c6f44b1b175f1a35e5e2dc1: Generated new TSK 0
I20260810 00:44:54.229008 29654 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.6.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab.
Entry for principal kudu/127.28.6.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab.
I20260810 00:44:54.278077 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.1:0
--local_ip_for_outbound_sockets=127.28.6.1
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:36471
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:46231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:44:54.622087 29677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:44:54.622351 29677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:44:54.622514 29677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:44:54.629508 29677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:44:54.629623 29677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:44:54.629797 29677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.1
I20260810 00:44:54.639124 29677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:46231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:36471
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.1
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:44:54.640277 29677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:44:54.641592 29677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:44:54.654598 29684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:44:54.654646 29685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:44:54.654678 29691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:44:54.655794 29677 server_base.cc:1034] running on GCE node
I20260810 00:44:54.656369 29677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:44:54.657243 29677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:44:54.658442 29677 hybrid_clock.cc:648] HybridClock initialized: now 1786322694658406 us; error 32 us; skew 500 ppm
I20260810 00:44:54.658761 29677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:44:54 dist-test-slave-1nl6 krb5kdc[29514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322694, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:44:54.661782 29677 init.cc:377] Logged in from keytab as kudu/127.28.6.1@KRBTEST.COM (short username kudu)
I20260810 00:44:54.663157 29677 webserver.cc:460] Webserver started at http://127.28.6.1:40269/ using document root <none> and password file <none>
I20260810 00:44:54.663872 29677 fs_manager.cc:359] Metadata directory not provided
I20260810 00:44:54.664019 29677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:44:54.664333 29677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:44:54.667172 29677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-0/data/instance:
uuid: "e4234d5abef54b2cae8ad2513c5b645b"
format_stamp: "Formatted at 2026-08-10 00:44:54 on dist-test-slave-1nl6"
server_key: "d67b7916cb1d85a743789f934b99b903"
server_key_iv: "940343d57f58110bb239638eabc5ec9f"
server_key_version: "encryptionkey@0"
I20260810 00:44:54.667964 29677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance:
uuid: "e4234d5abef54b2cae8ad2513c5b645b"
format_stamp: "Formatted at 2026-08-10 00:44:54 on dist-test-slave-1nl6"
server_key: "d67b7916cb1d85a743789f934b99b903"
server_key_iv: "940343d57f58110bb239638eabc5ec9f"
server_key_version: "encryptionkey@0"
I20260810 00:44:54.672869 29677 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 00:44:54.676371 29699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:44:54.677770 29677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 00:44:54.677937 29677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-0/wal
uuid: "e4234d5abef54b2cae8ad2513c5b645b"
format_stamp: "Formatted at 2026-08-10 00:44:54 on dist-test-slave-1nl6"
server_key: "d67b7916cb1d85a743789f934b99b903"
server_key_iv: "940343d57f58110bb239638eabc5ec9f"
server_key_version: "encryptionkey@0"
I20260810 00:44:54.678151 29677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:44:54.712934 29677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:44:54.717013 29677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:44:54.717321 29677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:44:54.718302 29677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:44:54.719653 29677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:44:54.719729 29677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:44:54.719913 29677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:44:54.719976 29677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:44:54.755352 29677 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.1:32951
I20260810 00:44:54.755403 29869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.1:32951 every 8 connection(s)
I20260810 00:44:54.756594 29677 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
I20260810 00:44:54.765664 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 29677
I20260810 00:44:54.765887 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance
I20260810 00:44:54.766637 28696 external_mini_cluster.cc:1428] Setting key fc51533ce137af8d6952b5b961b39329
Aug 10 00:44:54 dist-test-slave-1nl6 krb5kdc[29514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322694, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:44:54.778156 29872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:54.758999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:49853 (local address 127.28.6.62:36471)
0810 00:44:54.759491 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0810 00:44:54.759495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:44:54.760293 (+ 798us) server_negotiation.cc:407] Connection header received
0810 00:44:54.762135 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:44:54.762141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:44:54.762236 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:44:54.762360 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:44:54.763926 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:54.764655 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:54.766393 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:54.767063 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:54.767212 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:54.771225 (+ 4013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:44:54.771234 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:44:54.771242 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:44:54.771290 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:44:54.774719 (+ 3429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:54.775749 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:54.775753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:54.775755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:54.775906 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:54.776391 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:54.776394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:54.776397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:54.776713 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:44:54.776832 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 00:44:54.777716 (+ 884us) server_negotiation.cc:299] Negotiation successful
0810 00:44:54.777889 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":290,"threads_started":1}
I20260810 00:44:54.779680 29873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:54.759450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36471 (local address 127.28.6.1:49853)
0810 00:44:54.760135 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0810 00:44:54.760173 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 00:44:54.761853 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:44:54.762599 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:44:54.762606 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:44:54.763179 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:44:54.763752 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:44:54.763769 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:44:54.764842 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:44:54.764846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:44:54.766202 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:44:54.766219 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:44:54.767357 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:44:54.767368 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:44:54.767652 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:54.768405 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:44:54.768444 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:44:54.771008 (+ 2564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:44:54.774868 (+ 3860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:44:54.774879 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:44:54.774928 (+ 49us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:44:54.775627 (+ 699us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:44:54.776014 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:44:54.776021 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:44:54.776028 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:44:54.776279 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:44:54.776835 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:44:54.776846 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:44:54.777318 (+ 472us) client_negotiation.cc:769] Sending connection context
0810 00:44:54.777617 (+ 299us) client_negotiation.cc:240] Negotiation successful
0810 00:44:54.778184 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":290,"threads_started":1}
I20260810 00:44:54.781271 29870 heartbeater.cc:347] Connected to a master server at 127.28.6.62:36471
I20260810 00:44:54.781608 29870 heartbeater.cc:464] Registering TS with master...
I20260810 00:44:54.782398 29870 heartbeater.cc:511] Master 127.28.6.62:36471 requested a full tablet report, sending...
I20260810 00:44:54.784597 29580 ts_manager.cc:194] Registered new tserver with Master: e4234d5abef54b2cae8ad2513c5b645b (127.28.6.1:32951)
I20260810 00:44:54.785995 29580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.1@KRBTEST.COM'} at 127.28.6.1:49853
WARNING: no policy specified for kudu/127.28.6.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab.
Entry for principal kudu/127.28.6.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab.
I20260810 00:44:54.811053 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.2:0
--local_ip_for_outbound_sockets=127.28.6.2
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:36471
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:46231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:44:55.135670 29877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:44:55.136034 29877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:44:55.136222 29877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:44:55.145882 29877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:44:55.145972 29877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:44:55.146204 29877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.2
I20260810 00:44:55.155838 29877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:46231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:36471
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.2
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:44:55.156968 29877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:44:55.158280 29877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:44:55.171051 29886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:44:55.171245 29885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:44:55.171208 29889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:44:55.171957 29877 server_base.cc:1034] running on GCE node
I20260810 00:44:55.172574 29877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:44:55.173692 29877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:44:55.174944 29877 hybrid_clock.cc:648] HybridClock initialized: now 1786322695174873 us; error 27 us; skew 500 ppm
I20260810 00:44:55.175253 29877 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:44:55 dist-test-slave-1nl6 krb5kdc[29514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322695, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:44:55.178459 29877 init.cc:377] Logged in from keytab as kudu/127.28.6.2@KRBTEST.COM (short username kudu)
I20260810 00:44:55.179988 29877 webserver.cc:460] Webserver started at http://127.28.6.2:43065/ using document root <none> and password file <none>
I20260810 00:44:55.180650 29877 fs_manager.cc:359] Metadata directory not provided
I20260810 00:44:55.180795 29877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:44:55.181097 29877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:44:55.184185 29877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-1/data/instance:
uuid: "a85cfb0036124cd08ad5e507200b6a76"
format_stamp: "Formatted at 2026-08-10 00:44:55 on dist-test-slave-1nl6"
server_key: "8c6f2e75d2b6196d23895234752785e3"
server_key_iv: "dfb6893144c8d885dce20b7a243237e9"
server_key_version: "encryptionkey@0"
I20260810 00:44:55.184988 29877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance:
uuid: "a85cfb0036124cd08ad5e507200b6a76"
format_stamp: "Formatted at 2026-08-10 00:44:55 on dist-test-slave-1nl6"
server_key: "8c6f2e75d2b6196d23895234752785e3"
server_key_iv: "dfb6893144c8d885dce20b7a243237e9"
server_key_version: "encryptionkey@0"
I20260810 00:44:55.189858 29877 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260810 00:44:55.193207 29898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:44:55.194398 29877 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 00:44:55.194567 29877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-1/wal
uuid: "a85cfb0036124cd08ad5e507200b6a76"
format_stamp: "Formatted at 2026-08-10 00:44:55 on dist-test-slave-1nl6"
server_key: "8c6f2e75d2b6196d23895234752785e3"
server_key_iv: "dfb6893144c8d885dce20b7a243237e9"
server_key_version: "encryptionkey@0"
I20260810 00:44:55.194780 29877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:44:55.210394 29877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:44:55.213987 29877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:44:55.214250 29877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:44:55.215176 29877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:44:55.216377 29877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:44:55.216478 29877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:44:55.216616 29877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:44:55.216683 29877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:44:55.249465 29877 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.2:38567
I20260810 00:44:55.249475 30089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.2:38567 every 8 connection(s)
I20260810 00:44:55.250679 29877 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
I20260810 00:44:55.257295 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 29877
I20260810 00:44:55.257498 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance
I20260810 00:44:55.258020 28696 external_mini_cluster.cc:1428] Setting key a645045ff89c334709a3781e5f0dafc9
Aug 10 00:44:55 dist-test-slave-1nl6 krb5kdc[29514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322695, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:44:55.270541 29872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:55.253245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:40403 (local address 127.28.6.62:36471)
0810 00:44:55.253423 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 00:44:55.253428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:44:55.254568 (+ 1140us) server_negotiation.cc:407] Connection header received
0810 00:44:55.256317 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:44:55.256326 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:44:55.256395 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:44:55.256480 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:44:55.258396 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:55.259244 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:55.260749 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:55.261189 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:55.261333 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:55.264743 (+ 3410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:44:55.264752 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:44:55.264760 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:44:55.264807 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:44:55.267614 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:55.268352 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:55.268357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:55.268359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:55.268484 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:55.268918 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:55.268923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:55.268928 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:55.269187 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:44:55.269370 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0810 00:44:55.269986 (+ 616us) server_negotiation.cc:299] Negotiation successful
0810 00:44:55.270197 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 00:44:55.271486 30094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:55.253602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36471 (local address 127.28.6.2:40403)
0810 00:44:55.254364 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0810 00:44:55.254416 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 00:44:55.256078 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:44:55.256670 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:44:55.256678 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:44:55.257390 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:44:55.258163 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:44:55.258197 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:44:55.259405 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:44:55.259411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:44:55.260589 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:44:55.260602 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:44:55.261414 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:44:55.261422 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:44:55.261635 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:55.262302 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:44:55.262324 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:44:55.264518 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:44:55.267758 (+ 3240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:44:55.267767 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:44:55.267786 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:44:55.268203 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:44:55.268600 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:44:55.268606 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:44:55.268609 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:44:55.268792 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:44:55.269321 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:44:55.269330 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:44:55.269642 (+ 312us) client_negotiation.cc:769] Sending connection context
0810 00:44:55.269895 (+ 253us) client_negotiation.cc:240] Negotiation successful
0810 00:44:55.270350 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":286,"threads_started":1}
I20260810 00:44:55.273005 30091 heartbeater.cc:347] Connected to a master server at 127.28.6.62:36471
I20260810 00:44:55.273288 30091 heartbeater.cc:464] Registering TS with master...
I20260810 00:44:55.274030 30091 heartbeater.cc:511] Master 127.28.6.62:36471 requested a full tablet report, sending...
I20260810 00:44:55.275522 29580 ts_manager.cc:194] Registered new tserver with Master: a85cfb0036124cd08ad5e507200b6a76 (127.28.6.2:38567)
WARNING: no policy specified for kudu/127.28.6.3@KRBTEST.COM; defaulting to no policy
I20260810 00:44:55.276459 29580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.2@KRBTEST.COM'} at 127.28.6.2:40403
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab.
Entry for principal kudu/127.28.6.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab.
I20260810 00:44:55.300068 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.3:0
--local_ip_for_outbound_sockets=127.28.6.3
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:36471
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:46231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:44:55.620312 30100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:44:55.620555 30100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:44:55.620718 30100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:44:55.627688 30100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:44:55.627813 30100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:44:55.628003 30100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.3
I20260810 00:44:55.637302 30100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:46231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:36471
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.3
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:44:55.638468 30100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:44:55.639768 30100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:44:55.652999 30105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:44:55.653177 30108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:44:55.653028 30106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:44:55.654182 30100 server_base.cc:1034] running on GCE node
I20260810 00:44:55.654784 30100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:44:55.655683 30100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:44:55.656920 30100 hybrid_clock.cc:648] HybridClock initialized: now 1786322695656872 us; error 48 us; skew 500 ppm
I20260810 00:44:55.657251 30100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:44:55 dist-test-slave-1nl6 krb5kdc[29514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322695, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:44:55.660378 30100 init.cc:377] Logged in from keytab as kudu/127.28.6.3@KRBTEST.COM (short username kudu)
I20260810 00:44:55.662159 30100 webserver.cc:460] Webserver started at http://127.28.6.3:42859/ using document root <none> and password file <none>
I20260810 00:44:55.662748 30100 fs_manager.cc:359] Metadata directory not provided
I20260810 00:44:55.662846 30100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:44:55.663197 30100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:44:55.665944 30100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-2/data/instance:
uuid: "6252178320dd418982719931acc0a98b"
format_stamp: "Formatted at 2026-08-10 00:44:55 on dist-test-slave-1nl6"
server_key: "08dc82ad66b8291bdcfdfd58a7e20c2d"
server_key_iv: "455c65004b6503cd0613f65618631d5c"
server_key_version: "encryptionkey@0"
I20260810 00:44:55.666657 30100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance:
uuid: "6252178320dd418982719931acc0a98b"
format_stamp: "Formatted at 2026-08-10 00:44:55 on dist-test-slave-1nl6"
server_key: "08dc82ad66b8291bdcfdfd58a7e20c2d"
server_key_iv: "455c65004b6503cd0613f65618631d5c"
server_key_version: "encryptionkey@0"
I20260810 00:44:55.671584 30100 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 00:44:55.674798 30121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:44:55.676077 30100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260810 00:44:55.676239 30100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-2/wal
uuid: "6252178320dd418982719931acc0a98b"
format_stamp: "Formatted at 2026-08-10 00:44:55 on dist-test-slave-1nl6"
server_key: "08dc82ad66b8291bdcfdfd58a7e20c2d"
server_key_iv: "455c65004b6503cd0613f65618631d5c"
server_key_version: "encryptionkey@0"
I20260810 00:44:55.676427 30100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:44:55.699220 30100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:44:55.703299 30100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:44:55.703610 30100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:44:55.704649 30100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:44:55.705964 30100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:44:55.706039 30100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:44:55.706218 30100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:44:55.706269 30100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:44:55.739954 30100 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.3:45703
I20260810 00:44:55.740041 30286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.3:45703 every 8 connection(s)
I20260810 00:44:55.741220 30100 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
I20260810 00:44:55.745476 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 30100
I20260810 00:44:55.745669 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance
I20260810 00:44:55.746102 28696 external_mini_cluster.cc:1428] Setting key 22f6a8874c920331f6d7d7728dc82607
Aug 10 00:44:55 dist-test-slave-1nl6 krb5kdc[29514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322695, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:44:55.762019 29872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:55.743834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:57679 (local address 127.28.6.62:36471)
0810 00:44:55.744025 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 00:44:55.744030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:44:55.745175 (+ 1145us) server_negotiation.cc:407] Connection header received
0810 00:44:55.747399 (+ 2224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:44:55.747408 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:44:55.747494 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:44:55.747596 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:44:55.749547 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:55.750299 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:55.751837 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:55.752220 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:55.752322 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:55.756012 (+ 3690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:44:55.756017 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:44:55.756021 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:44:55.756052 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:44:55.758646 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:55.759561 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:55.759566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:55.759570 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:55.759742 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:55.760254 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:55.760258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:55.760260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:55.760628 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:44:55.760789 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 00:44:55.761482 (+ 693us) server_negotiation.cc:299] Negotiation successful
0810 00:44:55.761702 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 00:44:55.762841 30291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:55.744291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36471 (local address 127.28.6.3:57679)
0810 00:44:55.745007 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0810 00:44:55.745048 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 00:44:55.747060 (+ 2012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:44:55.747928 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:44:55.747940 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:44:55.748613 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:44:55.749370 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:44:55.749397 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:44:55.750430 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:44:55.750433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:44:55.751686 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:44:55.751698 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:44:55.752418 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:44:55.752423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:44:55.752615 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:55.753278 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:44:55.753300 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:44:55.755798 (+ 2498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:44:55.758783 (+ 2985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:44:55.758791 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:44:55.758804 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:44:55.759401 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:44:55.759887 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:44:55.759892 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:44:55.759895 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:44:55.760114 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:44:55.760763 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:44:55.760771 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:44:55.761076 (+ 305us) client_negotiation.cc:769] Sending connection context
0810 00:44:55.761348 (+ 272us) client_negotiation.cc:240] Negotiation successful
0810 00:44:55.761795 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":288,"threads_started":1}
I20260810 00:44:55.764086 30287 heartbeater.cc:347] Connected to a master server at 127.28.6.62:36471
I20260810 00:44:55.764384 30287 heartbeater.cc:464] Registering TS with master...
I20260810 00:44:55.765061 30287 heartbeater.cc:511] Master 127.28.6.62:36471 requested a full tablet report, sending...
I20260810 00:44:55.766422 29580 ts_manager.cc:194] Registered new tserver with Master: 6252178320dd418982719931acc0a98b (127.28.6.3:45703)
I20260810 00:44:55.767409 29580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.3@KRBTEST.COM'} at 127.28.6.3:57679
I20260810 00:44:55.773592 28696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 00:44:55.788803 29870 heartbeater.cc:503] Master 127.28.6.62:36471 was elected leader, sending a full tablet report...
I20260810 00:44:55.794401 29872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:55.780136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37738 (local address 127.28.6.62:36471)
0810 00:44:55.780334 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 00:44:55.780343 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:44:55.780793 (+ 450us) server_negotiation.cc:407] Connection header received
0810 00:44:55.780992 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:44:55.780998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:44:55.781079 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:44:55.781180 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:44:55.782776 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:55.784064 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:55.785281 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:55.785728 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:55.785879 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:55.788183 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:44:55.788190 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:44:55.788197 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:44:55.788235 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:44:55.791726 (+ 3491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:55.792362 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:55.792368 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:55.792372 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:55.792513 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:55.792899 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:55.792904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:55.792908 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:55.793219 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:44:55.793486 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0810 00:44:55.793875 (+ 389us) server_negotiation.cc:299] Negotiation successful
0810 00:44:55.794096 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 00:44:55.800465 29580 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 00:44:55.807253 29580 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 00:44:55.833053 30303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:55.818878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:32951 (local address 127.0.0.1:45548)
0810 00:44:55.820073 (+ 1195us) negotiation.cc:107] Waiting for socket to connect
0810 00:44:55.820113 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 00:44:55.820280 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:44:55.822075 (+ 1795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:44:55.822086 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:44:55.822117 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:44:55.822556 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:44:55.822576 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:44:55.825066 (+ 2490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:44:55.825071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:44:55.828617 (+ 3546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:44:55.828639 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:44:55.831708 (+ 3069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:44:55.831725 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:44:55.831926 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:55.831982 (+ 56us) client_negotiation.cc:769] Sending connection context
0810 00:44:55.832130 (+ 148us) client_negotiation.cc:240] Negotiation successful
0810 00:44:55.832346 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":941,"mutex_wait_us":44,"thread_start_us":1107,"threads_started":1}
I20260810 00:44:55.834620 30308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:55.819295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:45703 (local address 127.0.0.1:58990)
0810 00:44:55.821272 (+ 1977us) negotiation.cc:107] Waiting for socket to connect
0810 00:44:55.821292 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 00:44:55.821419 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:44:55.822328 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:44:55.822332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:44:55.822355 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:44:55.822861 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:44:55.822875 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:44:55.825407 (+ 2532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:44:55.825419 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:44:55.830675 (+ 5256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:44:55.830692 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:44:55.833568 (+ 2876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:44:55.833577 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:44:55.833825 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:55.833868 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 00:44:55.833954 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 00:44:55.834104 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1834,"thread_start_us":1694,"threads_started":1}
I20260810 00:44:55.835426 30306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:55.819207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45548 (local address 127.28.6.1:32951)
0810 00:44:55.821435 (+ 2228us) server_negotiation.cc:206] Beginning negotiation
0810 00:44:55.821443 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:44:55.821472 (+ 29us) server_negotiation.cc:407] Connection header received
0810 00:44:55.821575 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:44:55.821583 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:44:55.821834 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:44:55.822811 (+ 977us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:44:55.823076 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:55.824917 (+ 1841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:55.828799 (+ 3882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:55.830640 (+ 1841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:55.834106 (+ 3466us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:55.834409 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0810 00:44:55.834528 (+ 119us) server_negotiation.cc:299] Negotiation successful
0810 00:44:55.834751 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2070,"thread_start_us":1894,"threads_started":1}
I20260810 00:44:55.835907 30310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:55.819877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58990 (local address 127.28.6.3:45703)
0810 00:44:55.820928 (+ 1051us) server_negotiation.cc:206] Beginning negotiation
0810 00:44:55.820939 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 00:44:55.821541 (+ 602us) server_negotiation.cc:407] Connection header received
0810 00:44:55.821630 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:44:55.821636 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:44:55.821836 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:44:55.822030 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:44:55.823114 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:55.825193 (+ 2079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:55.830883 (+ 5690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:55.832766 (+ 1883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:55.832926 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:55.833357 (+ 431us) server_negotiation.cc:1035] Waiting for connection context
0810 00:44:55.835057 (+ 1700us) server_negotiation.cc:299] Negotiation successful
0810 00:44:55.835265 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":873,"thread_start_us":789,"threads_started":1}
I20260810 00:44:55.839669 29762 tablet_service.cc:1467] Processing CreateTablet for tablet cb06e1dc4b494aaf9dcc34f1003a757c (DEFAULT_TABLE table=test-table [id=c152fe3f12f443cda1afdb03ba850b69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:44:55.841786 29762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb06e1dc4b494aaf9dcc34f1003a757c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:44:55.842330 30194 tablet_service.cc:1467] Processing CreateTablet for tablet cb06e1dc4b494aaf9dcc34f1003a757c (DEFAULT_TABLE table=test-table [id=c152fe3f12f443cda1afdb03ba850b69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:44:55.844321 30194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb06e1dc4b494aaf9dcc34f1003a757c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:44:55.849794 30303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:55.821382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:38567 (local address 127.0.0.1:57142)
0810 00:44:55.833797 (+ 12415us) negotiation.cc:107] Waiting for socket to connect
0810 00:44:55.833816 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 00:44:55.833918 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:44:55.840384 (+ 6466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:44:55.840389 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:44:55.840416 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:44:55.840651 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:44:55.840675 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:44:55.843498 (+ 2823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:44:55.843504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:44:55.845909 (+ 2405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:44:55.845924 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:44:55.848744 (+ 2820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:44:55.848751 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:44:55.849014 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:55.849051 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 00:44:55.849139 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 00:44:55.849381 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12296,"thread_start_us":15351,"threads_started":1}
I20260810 00:44:55.851189 30312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:55.821726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57142 (local address 127.28.6.2:38567)
0810 00:44:55.838973 (+ 17247us) server_negotiation.cc:206] Beginning negotiation
0810 00:44:55.838995 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 00:44:55.839040 (+ 45us) server_negotiation.cc:407] Connection header received
0810 00:44:55.839147 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:44:55.839158 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:44:55.839474 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:44:55.839704 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:44:55.841019 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:55.843271 (+ 2252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:55.846175 (+ 2904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:55.848498 (+ 2323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:55.850086 (+ 1588us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:55.850341 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0810 00:44:55.850457 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 00:44:55.850703 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16967,"thread_start_us":18353,"threads_started":1}
I20260810 00:44:55.857455 29976 tablet_service.cc:1467] Processing CreateTablet for tablet cb06e1dc4b494aaf9dcc34f1003a757c (DEFAULT_TABLE table=test-table [id=c152fe3f12f443cda1afdb03ba850b69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:44:55.859295 29976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb06e1dc4b494aaf9dcc34f1003a757c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:44:55.865304 30313 tablet_bootstrap.cc:492] T cb06e1dc4b494aaf9dcc34f1003a757c P 6252178320dd418982719931acc0a98b: Bootstrap starting.
I20260810 00:44:55.872809 30314 tablet_bootstrap.cc:492] T cb06e1dc4b494aaf9dcc34f1003a757c P e4234d5abef54b2cae8ad2513c5b645b: Bootstrap starting.
I20260810 00:44:55.875761 30313 tablet_bootstrap.cc:654] T cb06e1dc4b494aaf9dcc34f1003a757c P 6252178320dd418982719931acc0a98b: Neither blocks nor log segments found. Creating new log.
I20260810 00:44:55.878564 30313 log.cc:824] T cb06e1dc4b494aaf9dcc34f1003a757c P 6252178320dd418982719931acc0a98b: Log is configured to *not* fsync() on all Append() calls
I20260810 00:44:55.879215 30315 tablet_bootstrap.cc:492] T cb06e1dc4b494aaf9dcc34f1003a757c P a85cfb0036124cd08ad5e507200b6a76: Bootstrap starting.
I20260810 00:44:55.881194 30314 tablet_bootstrap.cc:654] T cb06e1dc4b494aaf9dcc34f1003a757c P e4234d5abef54b2cae8ad2513c5b645b: Neither blocks nor log segments found. Creating new log.
I20260810 00:44:55.883342 30314 log.cc:824] T cb06e1dc4b494aaf9dcc34f1003a757c P e4234d5abef54b2cae8ad2513c5b645b: Log is configured to *not* fsync() on all Append() calls
I20260810 00:44:55.884527 30313 tablet_bootstrap.cc:492] T cb06e1dc4b494aaf9dcc34f1003a757c P 6252178320dd418982719931acc0a98b: No bootstrap required, opened a new log
I20260810 00:44:55.884933 30313 ts_tablet_manager.cc:1397] T cb06e1dc4b494aaf9dcc34f1003a757c P 6252178320dd418982719931acc0a98b: Time spent bootstrapping tablet: real 0.020s user 0.018s sys 0.000s
I20260810 00:44:55.885021 30315 tablet_bootstrap.cc:654] T cb06e1dc4b494aaf9dcc34f1003a757c P a85cfb0036124cd08ad5e507200b6a76: Neither blocks nor log segments found. Creating new log.
I20260810 00:44:55.886768 30315 log.cc:824] T cb06e1dc4b494aaf9dcc34f1003a757c P a85cfb0036124cd08ad5e507200b6a76: Log is configured to *not* fsync() on all Append() calls
I20260810 00:44:55.887470 30314 tablet_bootstrap.cc:492] T cb06e1dc4b494aaf9dcc34f1003a757c P e4234d5abef54b2cae8ad2513c5b645b: No bootstrap required, opened a new log
I20260810 00:44:55.887933 30314 ts_tablet_manager.cc:1397] T cb06e1dc4b494aaf9dcc34f1003a757c P e4234d5abef54b2cae8ad2513c5b645b: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.010s
I20260810 00:44:55.890429 30315 tablet_bootstrap.cc:492] T cb06e1dc4b494aaf9dcc34f1003a757c P a85cfb0036124cd08ad5e507200b6a76: No bootstrap required, opened a new log
I20260810 00:44:55.890750 30315 ts_tablet_manager.cc:1397] T cb06e1dc4b494aaf9dcc34f1003a757c P a85cfb0036124cd08ad5e507200b6a76: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260810 00:44:55.900058 30313 raft_consensus.cc:348] T cb06e1dc4b494aaf9dcc34f1003a757c P 6252178320dd418982719931acc0a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4234d5abef54b2cae8ad2513c5b645b" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 32951 } } peers { permanent_uuid: "6252178320dd418982719931acc0a98b" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45703 } } peers { permanent_uuid: "a85cfb0036124cd08ad5e507200b6a76" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 38567 } }
I20260810 00:44:55.900600 30313 raft_consensus.cc:374] T cb06e1dc4b494aaf9dcc34f1003a757c P 6252178320dd418982719931acc0a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:44:55.900801 30313 raft_consensus.cc:729] T cb06e1dc4b494aaf9dcc34f1003a757c P 6252178320dd418982719931acc0a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6252178320dd418982719931acc0a98b, State: Initialized, Role: FOLLOWER
I20260810 00:44:55.901629 30313 consensus_queue.cc:260] T cb06e1dc4b494aaf9dcc34f1003a757c P 6252178320dd418982719931acc0a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4234d5abef54b2cae8ad2513c5b645b" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 32951 } } peers { permanent_uuid: "6252178320dd418982719931acc0a98b" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45703 } } peers { permanent_uuid: "a85cfb0036124cd08ad5e507200b6a76" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 38567 } }
I20260810 00:44:55.902837 30315 raft_consensus.cc:348] T cb06e1dc4b494aaf9dcc34f1003a757c P a85cfb0036124cd08ad5e507200b6a76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4234d5abef54b2cae8ad2513c5b645b" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 32951 } } peers { permanent_uuid: "6252178320dd418982719931acc0a98b" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45703 } } peers { permanent_uuid: "a85cfb0036124cd08ad5e507200b6a76" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 38567 } }
I20260810 00:44:55.903463 30315 raft_consensus.cc:374] T cb06e1dc4b494aaf9dcc34f1003a757c P a85cfb0036124cd08ad5e507200b6a76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:44:55.903671 30315 raft_consensus.cc:729] T cb06e1dc4b494aaf9dcc34f1003a757c P a85cfb0036124cd08ad5e507200b6a76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a85cfb0036124cd08ad5e507200b6a76, State: Initialized, Role: FOLLOWER
I20260810 00:44:55.903883 30287 heartbeater.cc:503] Master 127.28.6.62:36471 was elected leader, sending a full tablet report...
I20260810 00:44:55.904129 30313 ts_tablet_manager.cc:1428] T cb06e1dc4b494aaf9dcc34f1003a757c P 6252178320dd418982719931acc0a98b: Time spent starting tablet: real 0.019s user 0.009s sys 0.009s
I20260810 00:44:55.904583 30314 raft_consensus.cc:348] T cb06e1dc4b494aaf9dcc34f1003a757c P e4234d5abef54b2cae8ad2513c5b645b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4234d5abef54b2cae8ad2513c5b645b" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 32951 } } peers { permanent_uuid: "6252178320dd418982719931acc0a98b" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45703 } } peers { permanent_uuid: "a85cfb0036124cd08ad5e507200b6a76" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 38567 } }
I20260810 00:44:55.904608 30315 consensus_queue.cc:260] T cb06e1dc4b494aaf9dcc34f1003a757c P a85cfb0036124cd08ad5e507200b6a76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4234d5abef54b2cae8ad2513c5b645b" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 32951 } } peers { permanent_uuid: "6252178320dd418982719931acc0a98b" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45703 } } peers { permanent_uuid: "a85cfb0036124cd08ad5e507200b6a76" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 38567 } }
I20260810 00:44:55.904960 30314 raft_consensus.cc:374] T cb06e1dc4b494aaf9dcc34f1003a757c P e4234d5abef54b2cae8ad2513c5b645b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:44:55.905051 30314 raft_consensus.cc:729] T cb06e1dc4b494aaf9dcc34f1003a757c P e4234d5abef54b2cae8ad2513c5b645b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4234d5abef54b2cae8ad2513c5b645b, State: Initialized, Role: FOLLOWER
I20260810 00:44:55.906090 30091 heartbeater.cc:503] Master 127.28.6.62:36471 was elected leader, sending a full tablet report...
I20260810 00:44:55.905895 30314 consensus_queue.cc:260] T cb06e1dc4b494aaf9dcc34f1003a757c P e4234d5abef54b2cae8ad2513c5b645b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4234d5abef54b2cae8ad2513c5b645b" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 32951 } } peers { permanent_uuid: "6252178320dd418982719931acc0a98b" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45703 } } peers { permanent_uuid: "a85cfb0036124cd08ad5e507200b6a76" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 38567 } }
I20260810 00:44:55.906878 30315 ts_tablet_manager.cc:1428] T cb06e1dc4b494aaf9dcc34f1003a757c P a85cfb0036124cd08ad5e507200b6a76: Time spent starting tablet: real 0.016s user 0.012s sys 0.001s
I20260810 00:44:55.909114 30314 ts_tablet_manager.cc:1428] T cb06e1dc4b494aaf9dcc34f1003a757c P e4234d5abef54b2cae8ad2513c5b645b: Time spent starting tablet: real 0.021s user 0.016s sys 0.002s
W20260810 00:44:55.992933 30289 tablet.cc:2369] T cb06e1dc4b494aaf9dcc34f1003a757c P 6252178320dd418982719931acc0a98b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 00:44:56.002677 30093 tablet.cc:2369] T cb06e1dc4b494aaf9dcc34f1003a757c P a85cfb0036124cd08ad5e507200b6a76: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 00:44:56.008920 29871 tablet.cc:2369] T cb06e1dc4b494aaf9dcc34f1003a757c P e4234d5abef54b2cae8ad2513c5b645b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:44:56.155835 30325 raft_consensus.cc:482] T cb06e1dc4b494aaf9dcc34f1003a757c P a85cfb0036124cd08ad5e507200b6a76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:44:56.156272 30325 raft_consensus.cc:504] T cb06e1dc4b494aaf9dcc34f1003a757c P a85cfb0036124cd08ad5e507200b6a76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4234d5abef54b2cae8ad2513c5b645b" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 32951 } } peers { permanent_uuid: "6252178320dd418982719931acc0a98b" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45703 } } peers { permanent_uuid: "a85cfb0036124cd08ad5e507200b6a76" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 38567 } }
I20260810 00:44:56.158536 30325 leader_election.cc:288] T cb06e1dc4b494aaf9dcc34f1003a757c P a85cfb0036124cd08ad5e507200b6a76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4234d5abef54b2cae8ad2513c5b645b (127.28.6.1:32951), 6252178320dd418982719931acc0a98b (127.28.6.3:45703)
I20260810 00:44:56.172127 30306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:56.159209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:34585 (local address 127.28.6.1:32951)
0810 00:44:56.159539 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0810 00:44:56.159549 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 00:44:56.160945 (+ 1396us) server_negotiation.cc:407] Connection header received
0810 00:44:56.161245 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:44:56.161250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:44:56.161345 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:44:56.161451 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:44:56.162863 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:56.164408 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:56.168756 (+ 4348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:56.170418 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:56.170609 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:56.170734 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 00:44:56.171473 (+ 739us) server_negotiation.cc:299] Negotiation successful
0810 00:44:56.171589 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260810 00:44:56.172297 30328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:56.158720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:32951 (local address 127.28.6.2:34585)
0810 00:44:56.160805 (+ 2085us) negotiation.cc:107] Waiting for socket to connect
0810 00:44:56.160829 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 00:44:56.161053 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:44:56.161485 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:44:56.161490 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:44:56.161519 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:44:56.162633 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:44:56.162654 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:44:56.164558 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:44:56.164563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:44:56.168566 (+ 4003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:44:56.168583 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:44:56.171042 (+ 2459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:44:56.171048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:44:56.171239 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:56.171265 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 00:44:56.171409 (+ 144us) client_negotiation.cc:240] Negotiation successful
0810 00:44:56.171543 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1894,"mutex_wait_us":123,"thread_start_us":1263,"threads_started":1}
I20260810 00:44:56.172384 30310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:56.159229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:51149 (local address 127.28.6.3:45703)
0810 00:44:56.159602 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0810 00:44:56.159607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:44:56.161069 (+ 1462us) server_negotiation.cc:407] Connection header received
0810 00:44:56.161270 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:44:56.161275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:44:56.161345 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:44:56.161451 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:44:56.162863 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:56.164433 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:56.168756 (+ 4323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:56.170506 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:56.170623 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:56.170799 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0810 00:44:56.171560 (+ 761us) server_negotiation.cc:299] Negotiation successful
0810 00:44:56.171726 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213}
I20260810 00:44:56.171650 30326 raft_consensus.cc:482] T cb06e1dc4b494aaf9dcc34f1003a757c P e4234d5abef54b2cae8ad2513c5b645b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:44:56.173398 30327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:56.158741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:45703 (local address 127.28.6.2:51149)
0810 00:44:56.160959 (+ 2218us) negotiation.cc:107] Waiting for socket to connect
0810 00:44:56.160975 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 00:44:56.161112 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:44:56.161485 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:44:56.161490 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:44:56.161597 (+ 107us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:44:56.162686 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:44:56.162699 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:44:56.164596 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:44:56.164603 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:44:56.168574 (+ 3971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:44:56.168583 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:44:56.171039 (+ 2456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:44:56.171046 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:44:56.171335 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:56.171374 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 00:44:56.172667 (+ 1293us) client_negotiation.cc:240] Negotiation successful
0810 00:44:56.172839 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2044,"mutex_wait_us":98,"spinlock_wait_cycles":23936,"thread_start_us":1713,"threads_started":1}
I20260810 00:44:56.172926 30326 raft_consensus.cc:504] T cb06e1dc4b494aaf9dcc34f1003a757c P e4234d5abef54b2cae8ad2513c5b645b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4234d5abef54b2cae8ad2513c5b645b" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 32951 } } peers { permanent_uuid: "6252178320dd418982719931acc0a98b" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45703 } } peers { permanent_uuid: "a85cfb0036124cd08ad5e507200b6a76" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 38567 } }
I20260810 00:44:56.174108 29795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb06e1dc4b494aaf9dcc34f1003a757c" candidate_uuid: "a85cfb0036124cd08ad5e507200b6a76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4234d5abef54b2cae8ad2513c5b645b" is_pre_election: true
I20260810 00:44:56.174790 29795 raft_consensus.cc:2454] T cb06e1dc4b494aaf9dcc34f1003a757c P e4234d5abef54b2cae8ad2513c5b645b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a85cfb0036124cd08ad5e507200b6a76 in term 0.
I20260810 00:44:56.174731 30222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb06e1dc4b494aaf9dcc34f1003a757c" candidate_uuid: "a85cfb0036124cd08ad5e507200b6a76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6252178320dd418982719931acc0a98b" is_pre_election: true
I20260810 00:44:56.175386 30222 raft_consensus.cc:2454] T cb06e1dc4b494aaf9dcc34f1003a757c P 6252178320dd418982719931acc0a98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a85cfb0036124cd08ad5e507200b6a76 in term 0.
I20260810 00:44:56.175834 29900 leader_election.cc:302] T cb06e1dc4b494aaf9dcc34f1003a757c P a85cfb0036124cd08ad5e507200b6a76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a85cfb0036124cd08ad5e507200b6a76, e4234d5abef54b2cae8ad2513c5b645b; no voters:
I20260810 00:44:56.176465 30325 raft_consensus.cc:2781] T cb06e1dc4b494aaf9dcc34f1003a757c P a85cfb0036124cd08ad5e507200b6a76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 00:44:56.176574 30326 leader_election.cc:288] T cb06e1dc4b494aaf9dcc34f1003a757c P e4234d5abef54b2cae8ad2513c5b645b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6252178320dd418982719931acc0a98b (127.28.6.3:45703), a85cfb0036124cd08ad5e507200b6a76 (127.28.6.2:38567)
I20260810 00:44:56.176728 30325 raft_consensus.cc:482] T cb06e1dc4b494aaf9dcc34f1003a757c P a85cfb0036124cd08ad5e507200b6a76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 00:44:56.176923 30325 raft_consensus.cc:3037] T cb06e1dc4b494aaf9dcc34f1003a757c P a85cfb0036124cd08ad5e507200b6a76 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:44:56.182722 30325 raft_consensus.cc:504] T cb06e1dc4b494aaf9dcc34f1003a757c P a85cfb0036124cd08ad5e507200b6a76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4234d5abef54b2cae8ad2513c5b645b" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 32951 } } peers { permanent_uuid: "6252178320dd418982719931acc0a98b" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45703 } } peers { permanent_uuid: "a85cfb0036124cd08ad5e507200b6a76" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 38567 } }
I20260810 00:44:56.183706 30325 leader_election.cc:288] T cb06e1dc4b494aaf9dcc34f1003a757c P a85cfb0036124cd08ad5e507200b6a76 [CANDIDATE]: Term 1 election: Requested vote from peers e4234d5abef54b2cae8ad2513c5b645b (127.28.6.1:32951), 6252178320dd418982719931acc0a98b (127.28.6.3:45703)
I20260810 00:44:56.184448 29795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb06e1dc4b494aaf9dcc34f1003a757c" candidate_uuid: "a85cfb0036124cd08ad5e507200b6a76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4234d5abef54b2cae8ad2513c5b645b"
I20260810 00:44:56.184613 30222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb06e1dc4b494aaf9dcc34f1003a757c" candidate_uuid: "a85cfb0036124cd08ad5e507200b6a76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6252178320dd418982719931acc0a98b"
I20260810 00:44:56.184736 29795 raft_consensus.cc:3037] T cb06e1dc4b494aaf9dcc34f1003a757c P e4234d5abef54b2cae8ad2513c5b645b [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:44:56.184885 30222 raft_consensus.cc:3037] T cb06e1dc4b494aaf9dcc34f1003a757c P 6252178320dd418982719931acc0a98b [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:44:56.189381 30222 raft_consensus.cc:2454] T cb06e1dc4b494aaf9dcc34f1003a757c P 6252178320dd418982719931acc0a98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a85cfb0036124cd08ad5e507200b6a76 in term 1.
I20260810 00:44:56.189605 29795 raft_consensus.cc:2454] T cb06e1dc4b494aaf9dcc34f1003a757c P e4234d5abef54b2cae8ad2513c5b645b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a85cfb0036124cd08ad5e507200b6a76 in term 1.
I20260810 00:44:56.190205 30329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:56.177118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:45703 (local address 127.28.6.1:39519)
0810 00:44:56.178317 (+ 1199us) negotiation.cc:107] Waiting for socket to connect
0810 00:44:56.178333 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 00:44:56.178529 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:44:56.179215 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:44:56.179220 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:44:56.179272 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:44:56.180621 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:44:56.180637 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:44:56.182610 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:44:56.182615 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:44:56.186063 (+ 3448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:44:56.186083 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:44:56.188693 (+ 2610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:44:56.188699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:44:56.188930 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:56.188964 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 00:44:56.189427 (+ 463us) client_negotiation.cc:240] Negotiation successful
0810 00:44:56.189603 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1033,"mutex_wait_us":3,"thread_start_us":717,"threads_started":1}
I20260810 00:44:56.190119 29902 leader_election.cc:302] T cb06e1dc4b494aaf9dcc34f1003a757c P a85cfb0036124cd08ad5e507200b6a76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6252178320dd418982719931acc0a98b, a85cfb0036124cd08ad5e507200b6a76; no voters:
I20260810 00:44:56.190560 30325 raft_consensus.cc:2781] T cb06e1dc4b494aaf9dcc34f1003a757c P a85cfb0036124cd08ad5e507200b6a76 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:44:56.190948 30310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:56.177279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:39519 (local address 127.28.6.3:45703)
0810 00:44:56.177436 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 00:44:56.177440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:44:56.178423 (+ 983us) server_negotiation.cc:407] Connection header received
0810 00:44:56.178725 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:44:56.178733 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:44:56.178853 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:44:56.178985 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:44:56.180786 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:56.182421 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:56.186296 (+ 3875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:56.188089 (+ 1793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:56.188229 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:56.188391 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 00:44:56.189119 (+ 728us) server_negotiation.cc:299] Negotiation successful
0810 00:44:56.189283 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 00:44:56.191068 30312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:56.179802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:42447 (local address 127.28.6.2:38567)
0810 00:44:56.180054 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0810 00:44:56.180059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:44:56.180967 (+ 908us) server_negotiation.cc:407] Connection header received
0810 00:44:56.181185 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:44:56.181192 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:44:56.181269 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:44:56.181370 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:44:56.182229 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:56.183906 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:56.187750 (+ 3844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:56.189424 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:56.189584 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:56.189682 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 00:44:56.190545 (+ 863us) server_negotiation.cc:299] Negotiation successful
0810 00:44:56.190695 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 00:44:56.191798 30325 raft_consensus.cc:686] T cb06e1dc4b494aaf9dcc34f1003a757c P a85cfb0036124cd08ad5e507200b6a76 [term 1 LEADER]: Becoming Leader. State: Replica: a85cfb0036124cd08ad5e507200b6a76, State: Running, Role: LEADER
I20260810 00:44:56.192185 30330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:56.178556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:38567 (local address 127.28.6.1:42447)
0810 00:44:56.180863 (+ 2307us) negotiation.cc:107] Waiting for socket to connect
0810 00:44:56.180886 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 00:44:56.181048 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:44:56.181401 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:44:56.181405 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:44:56.181432 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:44:56.181954 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:44:56.181963 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:44:56.184114 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:44:56.184119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:44:56.187509 (+ 3390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:44:56.187527 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:44:56.190038 (+ 2511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:44:56.190057 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:44:56.190345 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:56.190380 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 00:44:56.191470 (+ 1090us) client_negotiation.cc:240] Negotiation successful
0810 00:44:56.191632 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2127,"thread_start_us":1749,"threads_started":1}
I20260810 00:44:56.192160 30221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb06e1dc4b494aaf9dcc34f1003a757c" candidate_uuid: "e4234d5abef54b2cae8ad2513c5b645b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6252178320dd418982719931acc0a98b" is_pre_election: true
I20260810 00:44:56.192502 30325 consensus_queue.cc:237] T cb06e1dc4b494aaf9dcc34f1003a757c P a85cfb0036124cd08ad5e507200b6a76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4234d5abef54b2cae8ad2513c5b645b" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 32951 } } peers { permanent_uuid: "6252178320dd418982719931acc0a98b" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45703 } } peers { permanent_uuid: "a85cfb0036124cd08ad5e507200b6a76" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 38567 } }
I20260810 00:44:56.192773 30221 raft_consensus.cc:2379] T cb06e1dc4b494aaf9dcc34f1003a757c P 6252178320dd418982719931acc0a98b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4234d5abef54b2cae8ad2513c5b645b in current term 1: Already voted for candidate a85cfb0036124cd08ad5e507200b6a76 in this term.
I20260810 00:44:56.193501 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb06e1dc4b494aaf9dcc34f1003a757c" candidate_uuid: "e4234d5abef54b2cae8ad2513c5b645b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a85cfb0036124cd08ad5e507200b6a76" is_pre_election: true
I20260810 00:44:56.194983 29705 leader_election.cc:302] T cb06e1dc4b494aaf9dcc34f1003a757c P e4234d5abef54b2cae8ad2513c5b645b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4234d5abef54b2cae8ad2513c5b645b; no voters: 6252178320dd418982719931acc0a98b, a85cfb0036124cd08ad5e507200b6a76
I20260810 00:44:56.195675 30326 raft_consensus.cc:2726] T cb06e1dc4b494aaf9dcc34f1003a757c P e4234d5abef54b2cae8ad2513c5b645b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 00:44:56.198655 29579 catalog_manager.cc:5495] T cb06e1dc4b494aaf9dcc34f1003a757c P a85cfb0036124cd08ad5e507200b6a76 reported cstate change: term changed from 0 to 1, leader changed from <none> to a85cfb0036124cd08ad5e507200b6a76 (127.28.6.2). New cstate: current_term: 1 leader_uuid: "a85cfb0036124cd08ad5e507200b6a76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4234d5abef54b2cae8ad2513c5b645b" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 32951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6252178320dd418982719931acc0a98b" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a85cfb0036124cd08ad5e507200b6a76" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 38567 } health_report { overall_health: HEALTHY } } }
Aug 10 00:44:56 dist-test-slave-1nl6 krb5kdc[29514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.1@KRBTEST.COM
I20260810 00:44:56.226568 30306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:56.207199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45554 (local address 127.28.6.1:32951)
0810 00:44:56.207348 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 00:44:56.207353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:44:56.208874 (+ 1521us) server_negotiation.cc:407] Connection header received
0810 00:44:56.209073 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:44:56.209076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:44:56.209173 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:44:56.209253 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:44:56.211605 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:56.212527 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:56.213949 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:56.214494 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:56.214582 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:56.219664 (+ 5082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:44:56.219676 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:44:56.219691 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:44:56.219722 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:44:56.223699 (+ 3977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:56.224355 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:56.224363 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:56.224365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:56.224535 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:56.224973 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:56.224976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:56.224979 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:56.225415 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:44:56.225714 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0810 00:44:56.225989 (+ 275us) server_negotiation.cc:299] Negotiation successful
0810 00:44:56.226140 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 00:44:56 dist-test-slave-1nl6 krb5kdc[29514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322696, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 00:44:56 dist-test-slave-1nl6 krb5kdc[29514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322696, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.6.1@KRBTEST.COM
I20260810 00:44:56.285166 30306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:56.266279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45556 (local address 127.28.6.1:32951)
0810 00:44:56.266531 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0810 00:44:56.266540 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:44:56.267219 (+ 679us) server_negotiation.cc:407] Connection header received
0810 00:44:56.267393 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:44:56.267397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:44:56.267462 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:44:56.267546 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:44:56.269727 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:56.270826 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:56.272542 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:56.273184 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:56.273282 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:56.278671 (+ 5389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:44:56.278676 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:44:56.278680 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:44:56.278711 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:44:56.282349 (+ 3638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:56.283057 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:56.283063 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:56.283066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:56.283201 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:56.283600 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:56.283604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:56.283606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:56.283953 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:44:56.284300 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0810 00:44:56.284537 (+ 237us) server_negotiation.cc:299] Negotiation successful
0810 00:44:56.284743 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
Aug 10 00:44:56 dist-test-slave-1nl6 krb5kdc[29514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 00:44:56 dist-test-slave-1nl6 krb5kdc[29514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.1@KRBTEST.COM
I20260810 00:44:56.338016 30306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:56.319914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45566 (local address 127.28.6.1:32951)
0810 00:44:56.320204 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0810 00:44:56.320212 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:44:56.320603 (+ 391us) server_negotiation.cc:407] Connection header received
0810 00:44:56.320828 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:44:56.320832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:44:56.320899 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:44:56.321011 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:44:56.322806 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:56.323933 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:56.325471 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:56.326030 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:56.326199 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:56.331033 (+ 4834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:44:56.331038 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:44:56.331042 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:44:56.331075 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:44:56.334764 (+ 3689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:56.335629 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:56.335636 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:56.335639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:56.335776 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:56.336315 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:56.336334 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:56.336336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:56.336680 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:44:56.336960 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0810 00:44:56.337250 (+ 290us) server_negotiation.cc:299] Negotiation successful
0810 00:44:56.337528 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
W20260810 00:44:56.340392 29736 tablet_service.cc:2908] Rejecting scan request for tablet cb06e1dc4b494aaf9dcc34f1003a757c: Uninitialized: clean time has not yet been initialized
I20260810 00:44:56.343405 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 29677
I20260810 00:44:56.358084 30306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:56.344374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45578 (local address 127.28.6.1:32951)
0810 00:44:56.344541 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 00:44:56.344545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:44:56.344991 (+ 446us) server_negotiation.cc:407] Connection header received
0810 00:44:56.345123 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:44:56.345126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:44:56.345206 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:44:56.345284 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:44:56.346749 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:56.347633 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:56.349131 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:56.349665 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:56.349783 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:56.352823 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:44:56.352828 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:44:56.352831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:44:56.352890 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:44:56.355629 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:56.356267 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:56.356271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:56.356273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:56.356396 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:56.356790 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:56.356793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:56.356794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:56.357061 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:44:56.357451 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0810 00:44:56.357580 (+ 129us) server_negotiation.cc:299] Negotiation successful
0810 00:44:56.357749 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 00:44:56.358775 29865 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:44:56.451222 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 29677
I20260810 00:44:56.479313 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 29877
Aug 10 00:44:56 dist-test-slave-1nl6 krb5kdc[29514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.2@KRBTEST.COM
I20260810 00:44:56.493328 30312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:56.479996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57144 (local address 127.28.6.2:38567)
0810 00:44:56.480168 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 00:44:56.480173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:44:56.480191 (+ 18us) server_negotiation.cc:407] Connection header received
0810 00:44:56.480448 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:44:56.480451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:44:56.480520 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:44:56.480606 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:44:56.481799 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:56.482493 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:56.483863 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:56.484242 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:56.484339 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:56.488231 (+ 3892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:44:56.488243 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:44:56.488258 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:44:56.488296 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:44:56.490868 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:56.491540 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:56.491547 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:56.491550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:56.491682 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:56.492003 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:56.492006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:56.492008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:56.492385 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:44:56.492601 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0810 00:44:56.492839 (+ 238us) server_negotiation.cc:299] Negotiation successful
0810 00:44:56.492994 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 00:44:56.493984 30084 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:44:56.587705 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 29877
I20260810 00:44:56.612571 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 30100
Aug 10 00:44:56 dist-test-slave-1nl6 krb5kdc[29514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.3@KRBTEST.COM
I20260810 00:44:56.626801 30310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:56.613294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59000 (local address 127.28.6.3:45703)
0810 00:44:56.613478 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 00:44:56.613484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:44:56.613549 (+ 65us) server_negotiation.cc:407] Connection header received
0810 00:44:56.613776 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:44:56.613780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:44:56.613864 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:44:56.613971 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:44:56.615270 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:56.615960 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:56.617217 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:56.617629 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:56.617765 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:56.621754 (+ 3989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:44:56.621768 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:44:56.621787 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:44:56.621821 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:44:56.624537 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:56.625133 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:56.625140 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:56.625142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:56.625298 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:56.625617 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:56.625620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:56.625622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:56.625944 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:44:56.626162 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0810 00:44:56.626405 (+ 243us) server_negotiation.cc:299] Negotiation successful
0810 00:44:56.626567 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 00:44:56.627525 30281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:44:56.715009 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 30100
I20260810 00:44:56.739385 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 29535
I20260810 00:44:56.740581 29629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:44:56.846944 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 29535
2026-08-10T00:44:56Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4614 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:44:56 dist-test-slave-1nl6 krb5kdc[30359](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:44:56 dist-test-slave-1nl6 krb5kdc[30359](info): set up 2 sockets
Aug 10 00:44:56 dist-test-slave-1nl6 krb5kdc[30359](info): commencing operation
krb5kdc: starting...
W20260810 00:44:58.158556 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.259s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:44:58 dist-test-slave-1nl6 krb5kdc[30359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:44:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:44:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.6.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
I20260810 00:44:58.338171 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:40269
--webserver_interface=127.28.6.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:43879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:40269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:44:58.663496 30380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:44:58.663856 30380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:44:58.664021 30380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:44:58.671079 30380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:44:58.671228 30380 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 00:44:58.671324 30380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:44:58.671440 30380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:44:58.671542 30380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:44:58.671633 30380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:44:58.681381 30380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:43879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:40269
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:40269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:44:58.682724 30380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:44:58.684103 30380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:44:58.696187 30389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:44:58.696316 30388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:44:58.696162 30391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:44:58.697353 30380 server_base.cc:1034] running on GCE node
I20260810 00:44:58.698071 30380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:44:58.699498 30380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:44:58.700731 30380 hybrid_clock.cc:648] HybridClock initialized: now 1786322698700670 us; error 32 us; skew 500 ppm
I20260810 00:44:58.701071 30380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:44:58 dist-test-slave-1nl6 krb5kdc[30359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322698, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:44:58.704248 30380 init.cc:377] Logged in from keytab as kudu/127.28.6.62@KRBTEST.COM (short username kudu)
I20260810 00:44:58.705716 30380 webserver.cc:460] Webserver started at http://127.28.6.62:41279/ using document root <none> and password file <none>
I20260810 00:44:58.706409 30380 fs_manager.cc:359] Metadata directory not provided
I20260810 00:44:58.706523 30380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:44:58.706866 30380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:44:58.709810 30380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "c467b45029c6438eb2a518a4bf45137c"
format_stamp: "Formatted at 2026-08-10 00:44:58 on dist-test-slave-1nl6"
server_key: "51d359ea35f99e2faac86ed4b483f06d"
server_key_iv: "31e8a64d83786cd74a7d5cf80487b47c"
server_key_version: "encryptionkey@0"
I20260810 00:44:58.710597 30380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "c467b45029c6438eb2a518a4bf45137c"
format_stamp: "Formatted at 2026-08-10 00:44:58 on dist-test-slave-1nl6"
server_key: "51d359ea35f99e2faac86ed4b483f06d"
server_key_iv: "31e8a64d83786cd74a7d5cf80487b47c"
server_key_version: "encryptionkey@0"
I20260810 00:44:58.715756 30380 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.003s
I20260810 00:44:58.719203 30402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:44:58.720474 30380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 00:44:58.720649 30380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "c467b45029c6438eb2a518a4bf45137c"
format_stamp: "Formatted at 2026-08-10 00:44:58 on dist-test-slave-1nl6"
server_key: "51d359ea35f99e2faac86ed4b483f06d"
server_key_iv: "31e8a64d83786cd74a7d5cf80487b47c"
server_key_version: "encryptionkey@0"
I20260810 00:44:58.720855 30380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:44:58.744833 30380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:44:58.748836 30380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:44:58.749109 30380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:44:58.770629 30380 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:40269
I20260810 00:44:58.770637 30492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:40269 every 8 connection(s)
I20260810 00:44:58.772008 30380 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:44:58.774729 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 30380
I20260810 00:44:58.774995 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:44:58.775656 28696 external_mini_cluster.cc:1428] Setting key 7bf973c01fd3b40580e244fe9ea9da47
I20260810 00:44:58.775897 30493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:44:58.790340 30493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c467b45029c6438eb2a518a4bf45137c: Bootstrap starting.
Aug 10 00:44:58 dist-test-slave-1nl6 krb5kdc[30359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:44:58.793512 30493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c467b45029c6438eb2a518a4bf45137c: Neither blocks nor log segments found. Creating new log.
I20260810 00:44:58.794548 30493 log.cc:824] T 00000000000000000000000000000000 P c467b45029c6438eb2a518a4bf45137c: Log is configured to *not* fsync() on all Append() calls
I20260810 00:44:58.797410 30493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c467b45029c6438eb2a518a4bf45137c: No bootstrap required, opened a new log
I20260810 00:44:58.798861 30496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:58.777615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34666 (local address 127.28.6.62:40269)
0810 00:44:58.778404 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0810 00:44:58.778415 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 00:44:58.778457 (+ 42us) server_negotiation.cc:407] Connection header received
0810 00:44:58.780530 (+ 2073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:44:58.780551 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:44:58.780897 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:44:58.781208 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:44:58.783033 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:58.784305 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:58.786085 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:58.786777 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:58.786981 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:58.791761 (+ 4780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:44:58.791769 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:44:58.791781 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:44:58.791823 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:44:58.795044 (+ 3221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:58.795712 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:58.795726 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:58.795738 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:58.795897 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:58.796303 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:58.796308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:58.796312 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:58.796795 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:44:58.797110 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0810 00:44:58.797571 (+ 461us) server_negotiation.cc:299] Negotiation successful
0810 00:44:58.797970 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":371,"threads_started":1}
I20260810 00:44:58.805482 30493 raft_consensus.cc:348] T 00000000000000000000000000000000 P c467b45029c6438eb2a518a4bf45137c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c467b45029c6438eb2a518a4bf45137c" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 40269 } }
I20260810 00:44:58.805822 30493 raft_consensus.cc:374] T 00000000000000000000000000000000 P c467b45029c6438eb2a518a4bf45137c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:44:58.805936 30493 raft_consensus.cc:729] T 00000000000000000000000000000000 P c467b45029c6438eb2a518a4bf45137c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c467b45029c6438eb2a518a4bf45137c, State: Initialized, Role: FOLLOWER
I20260810 00:44:58.806473 30493 consensus_queue.cc:260] T 00000000000000000000000000000000 P c467b45029c6438eb2a518a4bf45137c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c467b45029c6438eb2a518a4bf45137c" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 40269 } }
I20260810 00:44:58.806722 30493 raft_consensus.cc:388] T 00000000000000000000000000000000 P c467b45029c6438eb2a518a4bf45137c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:44:58.806893 30493 raft_consensus.cc:482] T 00000000000000000000000000000000 P c467b45029c6438eb2a518a4bf45137c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:44:58.807111 30493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c467b45029c6438eb2a518a4bf45137c [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:44:58.809247 30493 raft_consensus.cc:504] T 00000000000000000000000000000000 P c467b45029c6438eb2a518a4bf45137c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c467b45029c6438eb2a518a4bf45137c" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 40269 } }
I20260810 00:44:58.809736 30493 leader_election.cc:302] T 00000000000000000000000000000000 P c467b45029c6438eb2a518a4bf45137c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c467b45029c6438eb2a518a4bf45137c; no voters:
I20260810 00:44:58.810279 30493 leader_election.cc:288] T 00000000000000000000000000000000 P c467b45029c6438eb2a518a4bf45137c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:44:58.810410 30498 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c467b45029c6438eb2a518a4bf45137c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:44:58.811168 30498 raft_consensus.cc:686] T 00000000000000000000000000000000 P c467b45029c6438eb2a518a4bf45137c [term 1 LEADER]: Becoming Leader. State: Replica: c467b45029c6438eb2a518a4bf45137c, State: Running, Role: LEADER
I20260810 00:44:58.811746 30498 consensus_queue.cc:237] T 00000000000000000000000000000000 P c467b45029c6438eb2a518a4bf45137c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c467b45029c6438eb2a518a4bf45137c" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 40269 } }
I20260810 00:44:58.812369 30493 sys_catalog.cc:564] T 00000000000000000000000000000000 P c467b45029c6438eb2a518a4bf45137c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:44:58.815073 30500 sys_catalog.cc:455] T 00000000000000000000000000000000 P c467b45029c6438eb2a518a4bf45137c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c467b45029c6438eb2a518a4bf45137c. Latest consensus state: current_term: 1 leader_uuid: "c467b45029c6438eb2a518a4bf45137c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c467b45029c6438eb2a518a4bf45137c" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 40269 } } }
I20260810 00:44:58.815086 30499 sys_catalog.cc:455] T 00000000000000000000000000000000 P c467b45029c6438eb2a518a4bf45137c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c467b45029c6438eb2a518a4bf45137c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c467b45029c6438eb2a518a4bf45137c" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 40269 } } }
I20260810 00:44:58.815589 30499 sys_catalog.cc:458] T 00000000000000000000000000000000 P c467b45029c6438eb2a518a4bf45137c [sys.catalog]: This master's current role is: LEADER
I20260810 00:44:58.815589 30500 sys_catalog.cc:458] T 00000000000000000000000000000000 P c467b45029c6438eb2a518a4bf45137c [sys.catalog]: This master's current role is: LEADER
I20260810 00:44:58.817459 30508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:44:58.821938 30508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:44:58.830562 30508 catalog_manager.cc:1347] Generated new cluster ID: b5886aa546d641dcb445605760b98961
I20260810 00:44:58.830680 30508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:44:58.846647 30508 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:44:58.847924 30508 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:44:58.858664 30508 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c467b45029c6438eb2a518a4bf45137c: Generated new TSK 0
I20260810 00:44:58.859493 30508 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.6.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab.
Entry for principal kudu/127.28.6.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab.
I20260810 00:44:58.918046 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.1:0
--local_ip_for_outbound_sockets=127.28.6.1
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:40269
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:43879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:44:59.245015 30528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:44:59.245309 30528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:44:59.245522 30528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:44:59.252713 30528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:44:59.252817 30528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:44:59.252985 30528 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 00:44:59.253113 30528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.1
I20260810 00:44:59.262655 30528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:43879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.6.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.6.62:40269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.1
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:44:59.263906 30528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:44:59.265244 30528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:44:59.278450 30534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:44:59.278424 30535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:44:59.278502 30538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:44:59.279302 30528 server_base.cc:1034] running on GCE node
I20260810 00:44:59.279893 30528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:44:59.280745 30528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:44:59.281985 30528 hybrid_clock.cc:648] HybridClock initialized: now 1786322699281904 us; error 60 us; skew 500 ppm
I20260810 00:44:59.282284 30528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:44:59 dist-test-slave-1nl6 krb5kdc[30359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322699, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:44:59.289980 30528 init.cc:377] Logged in from keytab as kudu/127.28.6.1@KRBTEST.COM (short username kudu)
I20260810 00:44:59.291589 30528 webserver.cc:460] Webserver started at http://127.28.6.1:33863/ using document root <none> and password file <none>
I20260810 00:44:59.292274 30528 fs_manager.cc:359] Metadata directory not provided
I20260810 00:44:59.292408 30528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:44:59.292730 30528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:44:59.295593 30528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-0/data/instance:
uuid: "a22ee065c94f479eb65b696fa8b44333"
format_stamp: "Formatted at 2026-08-10 00:44:59 on dist-test-slave-1nl6"
server_key: "36fd773162842466966346f16c2c5946"
server_key_iv: "e208a0648e52fd12394397249ef7bc8f"
server_key_version: "encryptionkey@0"
I20260810 00:44:59.296413 30528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance:
uuid: "a22ee065c94f479eb65b696fa8b44333"
format_stamp: "Formatted at 2026-08-10 00:44:59 on dist-test-slave-1nl6"
server_key: "36fd773162842466966346f16c2c5946"
server_key_iv: "e208a0648e52fd12394397249ef7bc8f"
server_key_version: "encryptionkey@0"
I20260810 00:44:59.301306 30528 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 00:44:59.304489 30547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:44:59.305716 30528 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 00:44:59.305989 30528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-0/wal
uuid: "a22ee065c94f479eb65b696fa8b44333"
format_stamp: "Formatted at 2026-08-10 00:44:59 on dist-test-slave-1nl6"
server_key: "36fd773162842466966346f16c2c5946"
server_key_iv: "e208a0648e52fd12394397249ef7bc8f"
server_key_version: "encryptionkey@0"
I20260810 00:44:59.306205 30528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:44:59.323590 30528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:44:59.327782 30528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:44:59.328054 30528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:44:59.330734 30528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:44:59.330839 30528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:44:59.331018 30528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:44:59.331153 30528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 00:44:59 dist-test-slave-1nl6 krb5kdc[30359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322699, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:44:59.355611 30560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:59.331241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:46337 (local address 127.28.6.62:40269)
0810 00:44:59.331720 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0810 00:44:59.331724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:44:59.332954 (+ 1230us) server_negotiation.cc:407] Connection header received
0810 00:44:59.335261 (+ 2307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:44:59.335266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:44:59.335374 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:44:59.335491 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:44:59.337028 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:59.338038 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:59.341535 (+ 3497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:59.341940 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:59.342042 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:59.346891 (+ 4849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:44:59.346916 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:44:59.346922 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:44:59.346967 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:44:59.350726 (+ 3759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:59.351826 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:59.351832 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:59.351836 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:59.352003 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:59.353462 (+ 1459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:59.353469 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:59.353474 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:59.353894 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:44:59.354056 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 00:44:59.354758 (+ 702us) server_negotiation.cc:299] Negotiation successful
0810 00:44:59.355018 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":266,"threads_started":1}
I20260810 00:44:59.356521 30563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:59.331907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:40269 (local address 127.28.6.1:46337)
0810 00:44:59.332800 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0810 00:44:59.332836 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 00:44:59.334862 (+ 2026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:44:59.335747 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:44:59.335755 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:44:59.336253 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:44:59.336784 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:44:59.336802 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:44:59.338264 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:44:59.338270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:44:59.341347 (+ 3077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:44:59.341368 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:44:59.342202 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:44:59.342212 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:44:59.342566 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:59.343544 (+ 978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:44:59.343591 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:44:59.346644 (+ 3053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:44:59.350909 (+ 4265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:44:59.350918 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:44:59.350959 (+ 41us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:44:59.351634 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:44:59.352122 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:44:59.352126 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:44:59.352130 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:44:59.353311 (+ 1181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:44:59.354041 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:44:59.354050 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:44:59.354410 (+ 360us) client_negotiation.cc:769] Sending connection context
0810 00:44:59.354654 (+ 244us) client_negotiation.cc:240] Negotiation successful
0810 00:44:59.355193 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":436,"threads_started":1}
I20260810 00:44:59.381312 30560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:59.363348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:39883 (local address 127.28.6.62:40269)
0810 00:44:59.363554 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 00:44:59.363563 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:44:59.364293 (+ 730us) server_negotiation.cc:407] Connection header received
0810 00:44:59.364525 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:44:59.364533 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:44:59.364628 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:44:59.364743 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:44:59.366428 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:59.367720 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:59.370078 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:59.370817 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:59.371019 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:59.373728 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:44:59.373738 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:44:59.373745 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:44:59.373794 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:44:59.377796 (+ 4002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:59.378775 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:59.378780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:59.378784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:59.378982 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:59.379359 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:59.379364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:59.379368 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:59.379796 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:44:59.379963 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 00:44:59.380548 (+ 585us) server_negotiation.cc:299] Negotiation successful
0810 00:44:59.380821 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 00:44:59.381546 30622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:59.363432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:40269 (local address 127.28.6.1:39883)
0810 00:44:59.364147 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0810 00:44:59.364170 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 00:44:59.364376 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:44:59.364810 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:44:59.364815 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:44:59.365643 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:44:59.366239 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:44:59.366260 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:44:59.367871 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:44:59.367879 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:44:59.369910 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:44:59.369930 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:44:59.371088 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:44:59.371096 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:44:59.371256 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:59.371951 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:44:59.371982 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:44:59.373484 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:44:59.377939 (+ 4455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:44:59.377947 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:44:59.377952 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:44:59.378642 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:44:59.379106 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:44:59.379109 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:44:59.379112 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:44:59.379240 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:44:59.379955 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:44:59.379963 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:44:59.380221 (+ 258us) client_negotiation.cc:769] Sending connection context
0810 00:44:59.380416 (+ 195us) client_negotiation.cc:240] Negotiation successful
0810 00:44:59.380843 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":375,"threads_started":1}
I20260810 00:44:59.387652 30528 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.1:44213
I20260810 00:44:59.387676 30718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.1:44213 every 8 connection(s)
I20260810 00:44:59.388813 30528 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
I20260810 00:44:59.390841 30719 heartbeater.cc:347] Connected to a master server at 127.28.6.62:40269
I20260810 00:44:59.391094 30719 heartbeater.cc:464] Registering TS with master...
I20260810 00:44:59.391731 30719 heartbeater.cc:511] Master 127.28.6.62:40269 requested a full tablet report, sending...
I20260810 00:44:59.393613 30424 ts_manager.cc:194] Registered new tserver with Master: a22ee065c94f479eb65b696fa8b44333 (127.28.6.1:44213)
I20260810 00:44:59.395085 30424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.1@KRBTEST.COM'} at 127.28.6.1:46337
I20260810 00:44:59.396471 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 30528
I20260810 00:44:59.396700 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance
I20260810 00:44:59.397519 28696 external_mini_cluster.cc:1428] Setting key 1cd75d1b48ae0e4cbc496cdb4606736c
WARNING: no policy specified for kudu/127.28.6.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab.
Entry for principal kudu/127.28.6.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab.
I20260810 00:44:59.437516 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.2:0
--local_ip_for_outbound_sockets=127.28.6.2
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:40269
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:43879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:44:59.760080 30725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:44:59.760355 30725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:44:59.760524 30725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:44:59.767639 30725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:44:59.767741 30725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:44:59.767918 30725 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 00:44:59.768045 30725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.2
I20260810 00:44:59.782301 30725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:43879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.6.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.6.62:40269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.2
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:44:59.783735 30725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:44:59.785434 30725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:44:59.800246 30735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:44:59.800314 30739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:44:59.800354 30741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:44:59.801019 30725 server_base.cc:1034] running on GCE node
I20260810 00:44:59.801566 30725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:44:59.802551 30725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:44:59.803808 30725 hybrid_clock.cc:648] HybridClock initialized: now 1786322699803749 us; error 51 us; skew 500 ppm
I20260810 00:44:59.804105 30725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:44:59 dist-test-slave-1nl6 krb5kdc[30359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322699, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:44:59.807333 30725 init.cc:377] Logged in from keytab as kudu/127.28.6.2@KRBTEST.COM (short username kudu)
I20260810 00:44:59.808636 30725 webserver.cc:460] Webserver started at http://127.28.6.2:33203/ using document root <none> and password file <none>
I20260810 00:44:59.809376 30725 fs_manager.cc:359] Metadata directory not provided
I20260810 00:44:59.809468 30725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:44:59.809752 30725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:44:59.812740 30725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-1/data/instance:
uuid: "e317b83358614dc6add668df09b881df"
format_stamp: "Formatted at 2026-08-10 00:44:59 on dist-test-slave-1nl6"
server_key: "94e7410dd8574a1bc1fb882c3d18b187"
server_key_iv: "97f18bd05110097d2a6ae8a2bba87ee5"
server_key_version: "encryptionkey@0"
I20260810 00:44:59.813495 30725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance:
uuid: "e317b83358614dc6add668df09b881df"
format_stamp: "Formatted at 2026-08-10 00:44:59 on dist-test-slave-1nl6"
server_key: "94e7410dd8574a1bc1fb882c3d18b187"
server_key_iv: "97f18bd05110097d2a6ae8a2bba87ee5"
server_key_version: "encryptionkey@0"
I20260810 00:44:59.818564 30725 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 00:44:59.821830 30750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:44:59.823204 30725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 00:44:59.823381 30725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-1/wal
uuid: "e317b83358614dc6add668df09b881df"
format_stamp: "Formatted at 2026-08-10 00:44:59 on dist-test-slave-1nl6"
server_key: "94e7410dd8574a1bc1fb882c3d18b187"
server_key_iv: "97f18bd05110097d2a6ae8a2bba87ee5"
server_key_version: "encryptionkey@0"
I20260810 00:44:59.823609 30725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:44:59.857817 30725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:44:59.861871 30725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:44:59.862161 30725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:44:59.864741 30725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:44:59.864858 30725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:44:59.865015 30725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:44:59.865090 30725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 00:44:59 dist-test-slave-1nl6 krb5kdc[30359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322699, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:44:59.887140 30560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:59.865517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:49683 (local address 127.28.6.62:40269)
0810 00:44:59.865686 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 00:44:59.865693 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:44:59.867179 (+ 1486us) server_negotiation.cc:407] Connection header received
0810 00:44:59.869299 (+ 2120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:44:59.869305 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:44:59.869406 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:44:59.869536 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:44:59.872372 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:59.873272 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:59.875673 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:59.876491 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:59.876646 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:59.881228 (+ 4582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:44:59.881233 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:44:59.881236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:44:59.881270 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:44:59.884134 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:59.884834 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:59.884846 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:59.884849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:59.884971 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:59.885440 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:59.885445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:59.885447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:59.885717 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:44:59.885874 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 00:44:59.886534 (+ 660us) server_negotiation.cc:299] Negotiation successful
0810 00:44:59.886754 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 00:44:59.888953 30760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:59.866081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:40269 (local address 127.28.6.2:49683)
0810 00:44:59.866996 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0810 00:44:59.867052 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 00:44:59.868949 (+ 1897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:44:59.870027 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:44:59.870039 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:44:59.871070 (+ 1031us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:44:59.872103 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:44:59.872143 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:44:59.873445 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:44:59.873449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:44:59.875499 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:44:59.875522 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:44:59.876737 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:44:59.876742 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:44:59.876910 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:59.877587 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:44:59.877614 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:44:59.880969 (+ 3355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:44:59.884278 (+ 3309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:44:59.884287 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:44:59.884302 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:44:59.884703 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:44:59.885095 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:44:59.885100 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:44:59.885103 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:44:59.885308 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:44:59.885848 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:44:59.885864 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:44:59.886186 (+ 322us) client_negotiation.cc:769] Sending connection context
0810 00:44:59.886442 (+ 256us) client_negotiation.cc:240] Negotiation successful
0810 00:44:59.886972 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":409,"threads_started":1}
I20260810 00:44:59.911669 30560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:59.894959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:57937 (local address 127.28.6.62:40269)
0810 00:44:59.895119 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 00:44:59.895122 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 00:44:59.896128 (+ 1006us) server_negotiation.cc:407] Connection header received
0810 00:44:59.896334 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:44:59.896337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:44:59.896405 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:44:59.896481 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:44:59.897882 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:59.898593 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:59.901257 (+ 2664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:44:59.901780 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:44:59.901944 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:59.903952 (+ 2008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:44:59.903961 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:44:59.903969 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:44:59.904018 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:44:59.908038 (+ 4020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:59.908985 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:59.908989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:59.908991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:59.909102 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:44:59.909689 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:44:59.909694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:44:59.909697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:44:59.910185 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:44:59.910337 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 00:44:59.910984 (+ 647us) server_negotiation.cc:299] Negotiation successful
0810 00:44:59.911248 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 00:44:59.911794 30853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:44:59.894960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:40269 (local address 127.28.6.2:57937)
0810 00:44:59.895999 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0810 00:44:59.896025 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 00:44:59.896199 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:44:59.896550 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:44:59.896559 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:44:59.897233 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:44:59.897741 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:44:59.897757 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:44:59.898742 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:44:59.898750 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:44:59.901089 (+ 2339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:44:59.901110 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:44:59.902024 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:44:59.902033 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:44:59.902328 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:44:59.903069 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:44:59.903086 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:44:59.903747 (+ 661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:44:59.908195 (+ 4448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:44:59.908205 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:44:59.908209 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:44:59.908786 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:44:59.909330 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:44:59.909338 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:44:59.909347 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:44:59.909549 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:44:59.910348 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:44:59.910355 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:44:59.910613 (+ 258us) client_negotiation.cc:769] Sending connection context
0810 00:44:59.910796 (+ 183us) client_negotiation.cc:240] Negotiation successful
0810 00:44:59.911190 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"thread_start_us":512,"threads_started":1}
I20260810 00:44:59.923321 30725 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.2:40727
I20260810 00:44:59.923348 30929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.2:40727 every 8 connection(s)
I20260810 00:44:59.924573 30725 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
I20260810 00:44:59.926692 30930 heartbeater.cc:347] Connected to a master server at 127.28.6.62:40269
I20260810 00:44:59.926950 30930 heartbeater.cc:464] Registering TS with master...
I20260810 00:44:59.927575 30930 heartbeater.cc:511] Master 127.28.6.62:40269 requested a full tablet report, sending...
I20260810 00:44:59.928913 30424 ts_manager.cc:194] Registered new tserver with Master: e317b83358614dc6add668df09b881df (127.28.6.2:40727)
I20260810 00:44:59.929663 30424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.2@KRBTEST.COM'} at 127.28.6.2:49683
I20260810 00:44:59.934736 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 30725
I20260810 00:44:59.934930 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance
I20260810 00:44:59.935309 28696 external_mini_cluster.cc:1428] Setting key becd6b27f27d6031ebd1a20617329bad
WARNING: no policy specified for kudu/127.28.6.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab.
Entry for principal kudu/127.28.6.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab.
I20260810 00:44:59.975243 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.3:0
--local_ip_for_outbound_sockets=127.28.6.3
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:40269
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:43879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:45:00.304800 30935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:00.305054 30935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:00.305147 30935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:00.312180 30935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:00.312305 30935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:00.312469 30935 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 00:45:00.312598 30935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.3
I20260810 00:45:00.322232 30935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:43879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.6.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.6.62:40269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.3
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:00.323539 30935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:00.324970 30935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:00.338109 30946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:00.338335 30947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:00.338359 30949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:00.339241 30935 server_base.cc:1034] running on GCE node
I20260810 00:45:00.339840 30935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:00.340727 30935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:00.341977 30935 hybrid_clock.cc:648] HybridClock initialized: now 1786322700341905 us; error 41 us; skew 500 ppm
I20260810 00:45:00.342272 30935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:45:00 dist-test-slave-1nl6 krb5kdc[30359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322700, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:00.345428 30935 init.cc:377] Logged in from keytab as kudu/127.28.6.3@KRBTEST.COM (short username kudu)
I20260810 00:45:00.346729 30935 webserver.cc:460] Webserver started at http://127.28.6.3:35057/ using document root <none> and password file <none>
I20260810 00:45:00.347447 30935 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:00.347577 30935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:00.347903 30935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:00.350682 30935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-2/data/instance:
uuid: "61d32e14a429452ba0597330033d3d79"
format_stamp: "Formatted at 2026-08-10 00:45:00 on dist-test-slave-1nl6"
server_key: "130b5b8bef25b351b494183e1362a35c"
server_key_iv: "5934f7175d0b3359777feba7ce02d4d5"
server_key_version: "encryptionkey@0"
I20260810 00:45:00.351738 30935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance:
uuid: "61d32e14a429452ba0597330033d3d79"
format_stamp: "Formatted at 2026-08-10 00:45:00 on dist-test-slave-1nl6"
server_key: "130b5b8bef25b351b494183e1362a35c"
server_key_iv: "5934f7175d0b3359777feba7ce02d4d5"
server_key_version: "encryptionkey@0"
I20260810 00:45:00.357511 30935 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 00:45:00.361160 30962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:00.362386 30935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 00:45:00.362569 30935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-2/wal
uuid: "61d32e14a429452ba0597330033d3d79"
format_stamp: "Formatted at 2026-08-10 00:45:00 on dist-test-slave-1nl6"
server_key: "130b5b8bef25b351b494183e1362a35c"
server_key_iv: "5934f7175d0b3359777feba7ce02d4d5"
server_key_version: "encryptionkey@0"
I20260810 00:45:00.362787 30935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:00.385885 30935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:00.390195 30935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:00.390477 30935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:00.393332 30935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:45:00.393445 30935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:00.393581 30935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:45:00.393658 30935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:00.398442 30719 heartbeater.cc:503] Master 127.28.6.62:40269 was elected leader, sending a full tablet report...
Aug 10 00:45:00 dist-test-slave-1nl6 krb5kdc[30359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322700, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:45:00.415853 30560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:00.394041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:49881 (local address 127.28.6.62:40269)
0810 00:45:00.394237 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:00.394244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:00.395719 (+ 1475us) server_negotiation.cc:407] Connection header received
0810 00:45:00.397734 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:00.397743 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:00.397845 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:00.398016 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:00.400148 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:00.401708 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:00.404295 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:00.404719 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:00.404853 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:00.408623 (+ 3770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:00.408633 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:00.408642 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:00.408691 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:00.412626 (+ 3935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:00.413434 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:00.413440 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:00.413443 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:00.413583 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:00.414054 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:00.414059 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:00.414063 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:00.414346 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:00.414497 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:00.415251 (+ 754us) server_negotiation.cc:299] Negotiation successful
0810 00:45:00.415472 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 00:45:00.416988 30976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:00.394571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:40269 (local address 127.28.6.3:49881)
0810 00:45:00.395550 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:00.395593 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:00.397453 (+ 1860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:00.398299 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:00.398315 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:00.399162 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:00.399949 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:00.399975 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:00.401900 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:00.401908 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:00.404130 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:00.404143 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:00.404955 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:00.404964 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:00.405162 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:00.405819 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:00.405845 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:00.408377 (+ 2532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:00.412805 (+ 4428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:00.412825 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:00.412840 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:00.413301 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:00.413723 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:00.413729 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:00.413733 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:00.413931 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:00.414476 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:00.414483 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:00.414804 (+ 321us) client_negotiation.cc:769] Sending connection context
0810 00:45:00.415122 (+ 318us) client_negotiation.cc:240] Negotiation successful
0810 00:45:00.415617 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":452,"threads_started":1}
I20260810 00:45:00.443392 30560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:00.425944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:56809 (local address 127.28.6.62:40269)
0810 00:45:00.426124 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:00.426131 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:00.426774 (+ 643us) server_negotiation.cc:407] Connection header received
0810 00:45:00.427070 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:00.427076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:00.427141 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:00.427245 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:00.428420 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:00.429624 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:00.432407 (+ 2783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:00.433177 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:00.433326 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:00.436055 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:00.436064 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:00.436070 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:00.436119 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:00.439646 (+ 3527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:00.440755 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:00.440762 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:00.440768 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:00.440951 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:00.441504 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:00.441511 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:00.441514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:00.441888 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:00.442063 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:00.442693 (+ 630us) server_negotiation.cc:299] Negotiation successful
0810 00:45:00.442972 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 00:45:00.443645 31060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:00.425954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:40269 (local address 127.28.6.3:56809)
0810 00:45:00.426651 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:00.426674 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:00.426889 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:00.427281 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:00.427285 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:00.428009 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:00.428284 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:00.428293 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:00.429807 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:00.429816 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:00.432171 (+ 2355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:00.432193 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:00.433394 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:00.433415 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:00.433664 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:00.434406 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:00.434434 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:00.435830 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:00.439810 (+ 3980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:00.439827 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:00.439838 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:00.440589 (+ 751us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:00.441094 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:00.441099 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:00.441103 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:00.441280 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:00.442035 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:00.442043 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:00.442333 (+ 290us) client_negotiation.cc:769] Sending connection context
0810 00:45:00.442527 (+ 194us) client_negotiation.cc:240] Negotiation successful
0810 00:45:00.442996 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":373,"threads_started":1}
I20260810 00:45:00.446233 30935 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.3:40003
I20260810 00:45:00.446318 31141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.3:40003 every 8 connection(s)
I20260810 00:45:00.447885 30935 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
I20260810 00:45:00.449822 31142 heartbeater.cc:347] Connected to a master server at 127.28.6.62:40269
I20260810 00:45:00.450048 31142 heartbeater.cc:464] Registering TS with master...
I20260810 00:45:00.450650 31142 heartbeater.cc:511] Master 127.28.6.62:40269 requested a full tablet report, sending...
I20260810 00:45:00.451990 30424 ts_manager.cc:194] Registered new tserver with Master: 61d32e14a429452ba0597330033d3d79 (127.28.6.3:40003)
I20260810 00:45:00.452579 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 30935
I20260810 00:45:00.452775 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance
I20260810 00:45:00.452816 30424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.3@KRBTEST.COM'} at 127.28.6.3:49881
I20260810 00:45:00.453320 28696 external_mini_cluster.cc:1428] Setting key 392171a1c50f997b9ebe321439488976
I20260810 00:45:00.456355 28696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 00:45:00 dist-test-slave-1nl6 krb5kdc[30359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322700, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 00:45:00 dist-test-slave-1nl6 krb5kdc[30359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322700, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:45:00.505313 30560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:00.491767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34668 (local address 127.28.6.62:40269)
0810 00:45:00.492067 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:00.492075 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:00.492388 (+ 313us) server_negotiation.cc:407] Connection header received
0810 00:45:00.492542 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:00.492546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:00.492613 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:00.492698 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:00.494065 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:00.494786 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:00.496135 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:00.496560 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:00.496657 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:00.500521 (+ 3864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:00.500526 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:00.500529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:00.500560 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:00.502876 (+ 2316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:00.503515 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:00.503519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:00.503521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:00.503653 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:00.504032 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:00.504035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:00.504037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:00.504407 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:00.504597 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:00.504840 (+ 243us) server_negotiation.cc:299] Negotiation successful
0810 00:45:00.504993 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260810 00:45:00.509356 30424 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 00:45:00.517360 30424 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 00:45:00.559981 31168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:00.534430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51568 (local address 127.28.6.2:40727)
0810 00:45:00.535521 (+ 1091us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:00.535533 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:00.547737 (+ 12204us) server_negotiation.cc:407] Connection header received
0810 00:45:00.547875 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:00.547886 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:00.548220 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:00.548542 (+ 322us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:00.550526 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:00.552661 (+ 2135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:00.556106 (+ 3445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:00.557247 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:00.557398 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:00.558347 (+ 949us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:00.559304 (+ 957us) server_negotiation.cc:299] Negotiation successful
0810 00:45:00.559511 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":842,"thread_start_us":595,"threads_started":1}
I20260810 00:45:00.560322 31162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:00.533290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:40727 (local address 127.0.0.1:51568)
0810 00:45:00.547334 (+ 14044us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:00.547359 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:00.547568 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:00.548833 (+ 1265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:00.548839 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:00.548880 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:00.550301 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:00.550326 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:00.552834 (+ 2508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:00.552846 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:00.555899 (+ 3053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:00.555912 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:00.557742 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:00.557764 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:00.558987 (+ 1223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:00.559054 (+ 67us) client_negotiation.cc:769] Sending connection context
0810 00:45:00.559214 (+ 160us) client_negotiation.cc:240] Negotiation successful
0810 00:45:00.559480 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13698,"mutex_wait_us":954,"thread_start_us":13158,"threads_started":1}
I20260810 00:45:00.563784 31160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:00.531469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:40003 (local address 127.0.0.1:46906)
0810 00:45:00.532495 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:00.532548 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:00.532743 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:00.539561 (+ 6818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:00.539569 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:00.539614 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:00.540547 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:00.540566 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:00.549945 (+ 9379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:00.549953 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:00.558760 (+ 8807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:00.558778 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:00.562313 (+ 3535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:00.562322 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:00.562564 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:00.562597 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 00:45:00.562711 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 00:45:00.562864 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"mutex_wait_us":2358,"thread_start_us":332,"threads_started":1}
I20260810 00:45:00.564167 31161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:00.531770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46906 (local address 127.28.6.3:40003)
0810 00:45:00.533104 (+ 1334us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:00.533114 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:00.533148 (+ 34us) server_negotiation.cc:407] Connection header received
0810 00:45:00.533267 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:00.533275 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:00.533605 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:00.533849 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:00.540840 (+ 6991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:00.549765 (+ 8925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:00.559048 (+ 9283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:00.561497 (+ 2449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:00.561594 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:00.561939 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:00.562783 (+ 844us) server_negotiation.cc:299] Negotiation successful
0810 00:45:00.563389 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1082,"thread_start_us":1261,"threads_started":1}
I20260810 00:45:00.564868 31163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:00.533717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:44213 (local address 127.0.0.1:43642)
0810 00:45:00.550586 (+ 16869us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:00.550605 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:00.550815 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:00.552119 (+ 1304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:00.552125 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:00.552168 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:00.554432 (+ 2264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:00.554464 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:00.556455 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:00.556462 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:00.560319 (+ 3857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:00.560337 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:00.563655 (+ 3318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:00.563666 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:00.563964 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:00.564008 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 00:45:00.564094 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 00:45:00.564214 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15362,"mutex_wait_us":1,"thread_start_us":15484,"threads_started":1}
I20260810 00:45:00.564882 30833 tablet_service.cc:1467] Processing CreateTablet for tablet d2e6addd0c85417da8eb837fa64d18eb (DEFAULT_TABLE table=test-table [id=cabadf43e65c48b0a94ce21dff1a2465]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:00.565800 31165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:00.533949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43642 (local address 127.28.6.1:44213)
0810 00:45:00.545880 (+ 11931us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:00.545892 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:00.551047 (+ 5155us) server_negotiation.cc:407] Connection header received
0810 00:45:00.551174 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:00.551182 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:00.551458 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:00.551653 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:00.554681 (+ 3028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:00.556252 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:00.560588 (+ 4336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:00.562880 (+ 2292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:00.563058 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:00.563325 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:00.565116 (+ 1791us) server_negotiation.cc:299] Negotiation successful
0810 00:45:00.565302 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11674,"thread_start_us":12580,"threads_started":1}
I20260810 00:45:00.566785 30833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2e6addd0c85417da8eb837fa64d18eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:00.568723 31041 tablet_service.cc:1467] Processing CreateTablet for tablet d2e6addd0c85417da8eb837fa64d18eb (DEFAULT_TABLE table=test-table [id=cabadf43e65c48b0a94ce21dff1a2465]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:00.570667 31041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2e6addd0c85417da8eb837fa64d18eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:00.572016 30626 tablet_service.cc:1467] Processing CreateTablet for tablet d2e6addd0c85417da8eb837fa64d18eb (DEFAULT_TABLE table=test-table [id=cabadf43e65c48b0a94ce21dff1a2465]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:00.573952 30626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2e6addd0c85417da8eb837fa64d18eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:00.584853 31169 tablet_bootstrap.cc:492] T d2e6addd0c85417da8eb837fa64d18eb P e317b83358614dc6add668df09b881df: Bootstrap starting.
I20260810 00:45:00.588583 31171 tablet_bootstrap.cc:492] T d2e6addd0c85417da8eb837fa64d18eb P 61d32e14a429452ba0597330033d3d79: Bootstrap starting.
I20260810 00:45:00.591893 31169 tablet_bootstrap.cc:654] T d2e6addd0c85417da8eb837fa64d18eb P e317b83358614dc6add668df09b881df: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:00.593641 31171 tablet_bootstrap.cc:654] T d2e6addd0c85417da8eb837fa64d18eb P 61d32e14a429452ba0597330033d3d79: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:00.593669 31169 log.cc:824] T d2e6addd0c85417da8eb837fa64d18eb P e317b83358614dc6add668df09b881df: Log is configured to *not* fsync() on all Append() calls
I20260810 00:45:00.596066 31171 log.cc:824] T d2e6addd0c85417da8eb837fa64d18eb P 61d32e14a429452ba0597330033d3d79: Log is configured to *not* fsync() on all Append() calls
I20260810 00:45:00.596160 31175 tablet_bootstrap.cc:492] T d2e6addd0c85417da8eb837fa64d18eb P a22ee065c94f479eb65b696fa8b44333: Bootstrap starting.
I20260810 00:45:00.598160 31169 tablet_bootstrap.cc:492] T d2e6addd0c85417da8eb837fa64d18eb P e317b83358614dc6add668df09b881df: No bootstrap required, opened a new log
I20260810 00:45:00.598507 31169 ts_tablet_manager.cc:1397] T d2e6addd0c85417da8eb837fa64d18eb P e317b83358614dc6add668df09b881df: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.005s
I20260810 00:45:00.601024 31171 tablet_bootstrap.cc:492] T d2e6addd0c85417da8eb837fa64d18eb P 61d32e14a429452ba0597330033d3d79: No bootstrap required, opened a new log
I20260810 00:45:00.601552 31171 ts_tablet_manager.cc:1397] T d2e6addd0c85417da8eb837fa64d18eb P 61d32e14a429452ba0597330033d3d79: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260810 00:45:00.605353 31175 tablet_bootstrap.cc:654] T d2e6addd0c85417da8eb837fa64d18eb P a22ee065c94f479eb65b696fa8b44333: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:00.607844 31175 log.cc:824] T d2e6addd0c85417da8eb837fa64d18eb P a22ee065c94f479eb65b696fa8b44333: Log is configured to *not* fsync() on all Append() calls
I20260810 00:45:00.611295 31169 raft_consensus.cc:348] T d2e6addd0c85417da8eb837fa64d18eb P e317b83358614dc6add668df09b881df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 } } peers { permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 } } peers { permanent_uuid: "a22ee065c94f479eb65b696fa8b44333" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44213 } }
I20260810 00:45:00.611956 31169 raft_consensus.cc:374] T d2e6addd0c85417da8eb837fa64d18eb P e317b83358614dc6add668df09b881df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:00.612205 31169 raft_consensus.cc:729] T d2e6addd0c85417da8eb837fa64d18eb P e317b83358614dc6add668df09b881df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e317b83358614dc6add668df09b881df, State: Initialized, Role: FOLLOWER
I20260810 00:45:00.613018 31175 tablet_bootstrap.cc:492] T d2e6addd0c85417da8eb837fa64d18eb P a22ee065c94f479eb65b696fa8b44333: No bootstrap required, opened a new log
I20260810 00:45:00.613087 31169 consensus_queue.cc:260] T d2e6addd0c85417da8eb837fa64d18eb P e317b83358614dc6add668df09b881df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 } } peers { permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 } } peers { permanent_uuid: "a22ee065c94f479eb65b696fa8b44333" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44213 } }
I20260810 00:45:00.613456 31175 ts_tablet_manager.cc:1397] T d2e6addd0c85417da8eb837fa64d18eb P a22ee065c94f479eb65b696fa8b44333: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.008s
I20260810 00:45:00.615886 30930 heartbeater.cc:503] Master 127.28.6.62:40269 was elected leader, sending a full tablet report...
I20260810 00:45:00.616779 31169 ts_tablet_manager.cc:1428] T d2e6addd0c85417da8eb837fa64d18eb P e317b83358614dc6add668df09b881df: Time spent starting tablet: real 0.018s user 0.012s sys 0.008s
I20260810 00:45:00.617820 31171 raft_consensus.cc:348] T d2e6addd0c85417da8eb837fa64d18eb P 61d32e14a429452ba0597330033d3d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 } } peers { permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 } } peers { permanent_uuid: "a22ee065c94f479eb65b696fa8b44333" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44213 } }
I20260810 00:45:00.618480 31171 raft_consensus.cc:374] T d2e6addd0c85417da8eb837fa64d18eb P 61d32e14a429452ba0597330033d3d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:00.618672 31171 raft_consensus.cc:729] T d2e6addd0c85417da8eb837fa64d18eb P 61d32e14a429452ba0597330033d3d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61d32e14a429452ba0597330033d3d79, State: Initialized, Role: FOLLOWER
I20260810 00:45:00.619505 31171 consensus_queue.cc:260] T d2e6addd0c85417da8eb837fa64d18eb P 61d32e14a429452ba0597330033d3d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 } } peers { permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 } } peers { permanent_uuid: "a22ee065c94f479eb65b696fa8b44333" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44213 } }
I20260810 00:45:00.620921 31142 heartbeater.cc:503] Master 127.28.6.62:40269 was elected leader, sending a full tablet report...
I20260810 00:45:00.621548 31171 ts_tablet_manager.cc:1428] T d2e6addd0c85417da8eb837fa64d18eb P 61d32e14a429452ba0597330033d3d79: Time spent starting tablet: real 0.020s user 0.018s sys 0.003s
I20260810 00:45:00.622310 31179 raft_consensus.cc:482] T d2e6addd0c85417da8eb837fa64d18eb P e317b83358614dc6add668df09b881df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:45:00.623245 31179 raft_consensus.cc:504] T d2e6addd0c85417da8eb837fa64d18eb P e317b83358614dc6add668df09b881df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 } } peers { permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 } } peers { permanent_uuid: "a22ee065c94f479eb65b696fa8b44333" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44213 } }
I20260810 00:45:00.630378 31175 raft_consensus.cc:348] T d2e6addd0c85417da8eb837fa64d18eb P a22ee065c94f479eb65b696fa8b44333 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 } } peers { permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 } } peers { permanent_uuid: "a22ee065c94f479eb65b696fa8b44333" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44213 } }
I20260810 00:45:00.630859 31179 leader_election.cc:288] T d2e6addd0c85417da8eb837fa64d18eb P e317b83358614dc6add668df09b881df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61d32e14a429452ba0597330033d3d79 (127.28.6.3:40003), a22ee065c94f479eb65b696fa8b44333 (127.28.6.1:44213)
I20260810 00:45:00.631014 31175 raft_consensus.cc:374] T d2e6addd0c85417da8eb837fa64d18eb P a22ee065c94f479eb65b696fa8b44333 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:00.631232 31175 raft_consensus.cc:729] T d2e6addd0c85417da8eb837fa64d18eb P a22ee065c94f479eb65b696fa8b44333 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a22ee065c94f479eb65b696fa8b44333, State: Initialized, Role: FOLLOWER
I20260810 00:45:00.632112 31175 consensus_queue.cc:260] T d2e6addd0c85417da8eb837fa64d18eb P a22ee065c94f479eb65b696fa8b44333 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 } } peers { permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 } } peers { permanent_uuid: "a22ee065c94f479eb65b696fa8b44333" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44213 } }
I20260810 00:45:00.634913 31175 ts_tablet_manager.cc:1428] T d2e6addd0c85417da8eb837fa64d18eb P a22ee065c94f479eb65b696fa8b44333: Time spent starting tablet: real 0.021s user 0.011s sys 0.008s
I20260810 00:45:00.639268 31165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:00.627170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:56217 (local address 127.28.6.1:44213)
0810 00:45:00.627428 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:00.627439 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:00.628904 (+ 1465us) server_negotiation.cc:407] Connection header received
0810 00:45:00.629258 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:00.629265 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:00.629388 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:00.629453 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:00.631292 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:00.632692 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:00.636051 (+ 3359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:00.637561 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:00.637730 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:00.637898 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:00.638676 (+ 778us) server_negotiation.cc:299] Negotiation successful
0810 00:45:00.638841 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 00:45:00.639464 31183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:00.626944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:44213 (local address 127.28.6.2:56217)
0810 00:45:00.628800 (+ 1856us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:00.628822 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:00.629039 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:00.629822 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:00.629826 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:00.629913 (+ 87us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:00.630983 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:00.631000 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:00.632917 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:00.632926 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:00.635854 (+ 2928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:00.635867 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:00.638302 (+ 2435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:00.638312 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:00.638482 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:00.638507 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 00:45:00.638633 (+ 126us) client_negotiation.cc:240] Negotiation successful
0810 00:45:00.638836 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1682,"thread_start_us":3159,"threads_started":1}
I20260810 00:45:00.640736 31161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:00.627848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:35327 (local address 127.28.6.3:40003)
0810 00:45:00.628067 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:00.628074 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:00.631230 (+ 3156us) server_negotiation.cc:407] Connection header received
0810 00:45:00.631535 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:00.631539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:00.631616 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:00.631709 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:00.633007 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:00.634447 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:00.637429 (+ 2982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:00.638997 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:00.639136 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:00.639261 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:00.640222 (+ 961us) server_negotiation.cc:299] Negotiation successful
0810 00:45:00.640348 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 00:45:00.640515 30654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2e6addd0c85417da8eb837fa64d18eb" candidate_uuid: "e317b83358614dc6add668df09b881df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a22ee065c94f479eb65b696fa8b44333" is_pre_election: true
W20260810 00:45:00.641170 30720 tablet.cc:2369] T d2e6addd0c85417da8eb837fa64d18eb P a22ee065c94f479eb65b696fa8b44333: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:45:00.641254 30654 raft_consensus.cc:2454] T d2e6addd0c85417da8eb837fa64d18eb P a22ee065c94f479eb65b696fa8b44333 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e317b83358614dc6add668df09b881df in term 0.
I20260810 00:45:00.641841 31181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:00.627047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:40003 (local address 127.28.6.2:35327)
0810 00:45:00.631149 (+ 4102us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:00.631166 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:00.631350 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:00.631925 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:00.631928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:00.631948 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:00.632627 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:00.632645 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:00.634695 (+ 2050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:00.634700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:00.637234 (+ 2534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:00.637248 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:00.639788 (+ 2540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:00.639795 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:00.640053 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:00.640077 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 00:45:00.640191 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 00:45:00.640316 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3939,"thread_start_us":1162,"threads_started":1}
I20260810 00:45:00.642127 30755 leader_election.cc:302] T d2e6addd0c85417da8eb837fa64d18eb P e317b83358614dc6add668df09b881df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a22ee065c94f479eb65b696fa8b44333, e317b83358614dc6add668df09b881df; no voters:
I20260810 00:45:00.642959 31179 raft_consensus.cc:2781] T d2e6addd0c85417da8eb837fa64d18eb P e317b83358614dc6add668df09b881df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 00:45:00.643257 31179 raft_consensus.cc:482] T d2e6addd0c85417da8eb837fa64d18eb P e317b83358614dc6add668df09b881df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 00:45:00.643517 31179 raft_consensus.cc:3037] T d2e6addd0c85417da8eb837fa64d18eb P e317b83358614dc6add668df09b881df [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:00.643424 31075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2e6addd0c85417da8eb837fa64d18eb" candidate_uuid: "e317b83358614dc6add668df09b881df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61d32e14a429452ba0597330033d3d79" is_pre_election: true
I20260810 00:45:00.644443 31075 raft_consensus.cc:2454] T d2e6addd0c85417da8eb837fa64d18eb P 61d32e14a429452ba0597330033d3d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e317b83358614dc6add668df09b881df in term 0.
I20260810 00:45:00.647426 31179 raft_consensus.cc:504] T d2e6addd0c85417da8eb837fa64d18eb P e317b83358614dc6add668df09b881df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 } } peers { permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 } } peers { permanent_uuid: "a22ee065c94f479eb65b696fa8b44333" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44213 } }
I20260810 00:45:00.648200 31179 leader_election.cc:288] T d2e6addd0c85417da8eb837fa64d18eb P e317b83358614dc6add668df09b881df [CANDIDATE]: Term 1 election: Requested vote from peers 61d32e14a429452ba0597330033d3d79 (127.28.6.3:40003), a22ee065c94f479eb65b696fa8b44333 (127.28.6.1:44213)
I20260810 00:45:00.648590 31075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2e6addd0c85417da8eb837fa64d18eb" candidate_uuid: "e317b83358614dc6add668df09b881df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61d32e14a429452ba0597330033d3d79"
I20260810 00:45:00.648838 31075 raft_consensus.cc:3037] T d2e6addd0c85417da8eb837fa64d18eb P 61d32e14a429452ba0597330033d3d79 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:00.648823 30654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2e6addd0c85417da8eb837fa64d18eb" candidate_uuid: "e317b83358614dc6add668df09b881df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a22ee065c94f479eb65b696fa8b44333"
I20260810 00:45:00.649076 30654 raft_consensus.cc:3037] T d2e6addd0c85417da8eb837fa64d18eb P a22ee065c94f479eb65b696fa8b44333 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:00.653558 30654 raft_consensus.cc:2454] T d2e6addd0c85417da8eb837fa64d18eb P a22ee065c94f479eb65b696fa8b44333 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e317b83358614dc6add668df09b881df in term 1.
I20260810 00:45:00.653578 31075 raft_consensus.cc:2454] T d2e6addd0c85417da8eb837fa64d18eb P 61d32e14a429452ba0597330033d3d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e317b83358614dc6add668df09b881df in term 1.
I20260810 00:45:00.654136 30751 leader_election.cc:302] T d2e6addd0c85417da8eb837fa64d18eb P e317b83358614dc6add668df09b881df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 61d32e14a429452ba0597330033d3d79, a22ee065c94f479eb65b696fa8b44333, e317b83358614dc6add668df09b881df; no voters:
I20260810 00:45:00.654675 31179 raft_consensus.cc:2781] T d2e6addd0c85417da8eb837fa64d18eb P e317b83358614dc6add668df09b881df [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:45:00.655414 31179 raft_consensus.cc:686] T d2e6addd0c85417da8eb837fa64d18eb P e317b83358614dc6add668df09b881df [term 1 LEADER]: Becoming Leader. State: Replica: e317b83358614dc6add668df09b881df, State: Running, Role: LEADER
I20260810 00:45:00.656059 31179 consensus_queue.cc:237] T d2e6addd0c85417da8eb837fa64d18eb P e317b83358614dc6add668df09b881df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 } } peers { permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 } } peers { permanent_uuid: "a22ee065c94f479eb65b696fa8b44333" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44213 } }
I20260810 00:45:00.660590 30423 catalog_manager.cc:5495] T d2e6addd0c85417da8eb837fa64d18eb P e317b83358614dc6add668df09b881df reported cstate change: term changed from 0 to 1, leader changed from <none> to e317b83358614dc6add668df09b881df (127.28.6.2). New cstate: current_term: 1 leader_uuid: "e317b83358614dc6add668df09b881df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a22ee065c94f479eb65b696fa8b44333" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44213 } health_report { overall_health: UNKNOWN } } }
W20260810 00:45:00.676566 30931 tablet.cc:2369] T d2e6addd0c85417da8eb837fa64d18eb P e317b83358614dc6add668df09b881df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:45:00.695340 31168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:00.688043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51572 (local address 127.28.6.2:40727)
0810 00:45:00.688292 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:00.688299 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:00.688610 (+ 311us) server_negotiation.cc:407] Connection header received
0810 00:45:00.688941 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:00.688945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:00.689038 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:00.689181 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:45:00.689763 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:00.690752 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:00.692417 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:00.693004 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:00.693152 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:00.693872 (+ 720us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:45:00.694526 (+ 654us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:45:00.694683 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:00.694829 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 00:45:00.694964 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260810 00:45:00.699620 31143 tablet.cc:2369] T d2e6addd0c85417da8eb837fa64d18eb P 61d32e14a429452ba0597330033d3d79: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:45:00.705545 30654 raft_consensus.cc:1261] T d2e6addd0c85417da8eb837fa64d18eb P a22ee065c94f479eb65b696fa8b44333 [term 1 FOLLOWER]: Refusing update from remote peer e317b83358614dc6add668df09b881df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:45:00.705804 31075 raft_consensus.cc:1261] T d2e6addd0c85417da8eb837fa64d18eb P 61d32e14a429452ba0597330033d3d79 [term 1 FOLLOWER]: Refusing update from remote peer e317b83358614dc6add668df09b881df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:45:00.706884 31190 consensus_queue.cc:1035] T d2e6addd0c85417da8eb837fa64d18eb P e317b83358614dc6add668df09b881df [LEADER]: Connected to new peer: Peer: permanent_uuid: "a22ee065c94f479eb65b696fa8b44333" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:45:00.707401 31179 consensus_queue.cc:1035] T d2e6addd0c85417da8eb837fa64d18eb P e317b83358614dc6add668df09b881df [LEADER]: Connected to new peer: Peer: permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:45:00.724596 31197 mvcc.cc:205] Tried to move back new op lower bound from 7316777782078398464 to 7316777781891465216. Current Snapshot: MvccSnapshot[applied={T|T < 7316777782078398464}]
I20260810 00:45:00.747121 30424 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:45:00.747699 30424 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:45:00.752620 30424 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c467b45029c6438eb2a518a4bf45137c: Sending DeleteTablet for 3 replicas of tablet d2e6addd0c85417da8eb837fa64d18eb
I20260810 00:45:00.754321 31041 tablet_service.cc:1514] Processing DeleteTablet for tablet d2e6addd0c85417da8eb837fa64d18eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:45:00 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:46906
I20260810 00:45:00.754769 30833 tablet_service.cc:1514] Processing DeleteTablet for tablet d2e6addd0c85417da8eb837fa64d18eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:45:00 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:51568
I20260810 00:45:00.755017 30626 tablet_service.cc:1514] Processing DeleteTablet for tablet d2e6addd0c85417da8eb837fa64d18eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:45:00 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:43642
I20260810 00:45:00.758450 30423 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 00:45:00.759519 30423 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 00:45:00.773123 31217 tablet_replica.cc:330] T d2e6addd0c85417da8eb837fa64d18eb P 61d32e14a429452ba0597330033d3d79: stopping tablet replica
I20260810 00:45:00.773749 31217 raft_consensus.cc:2229] T d2e6addd0c85417da8eb837fa64d18eb P 61d32e14a429452ba0597330033d3d79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:45:00.774336 31217 raft_consensus.cc:2258] T d2e6addd0c85417da8eb837fa64d18eb P 61d32e14a429452ba0597330033d3d79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:45:00.776098 30625 tablet_service.cc:1467] Processing CreateTablet for tablet 0be5926e18d943cc90237cf719aa958e (DEFAULT_TABLE table=test-table [id=e318786d97b94b37b61ba08e4567c2a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:00.777050 31217 ts_tablet_manager.cc:1905] T d2e6addd0c85417da8eb837fa64d18eb P 61d32e14a429452ba0597330033d3d79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:45:00.777184 30625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0be5926e18d943cc90237cf719aa958e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:00.785740 31041 tablet_service.cc:1467] Processing CreateTablet for tablet 0be5926e18d943cc90237cf719aa958e (DEFAULT_TABLE table=test-table [id=e318786d97b94b37b61ba08e4567c2a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:00.786469 31041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0be5926e18d943cc90237cf719aa958e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:00.789155 30831 tablet_service.cc:1467] Processing CreateTablet for tablet 0be5926e18d943cc90237cf719aa958e (DEFAULT_TABLE table=test-table [id=e318786d97b94b37b61ba08e4567c2a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:00.790447 30831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0be5926e18d943cc90237cf719aa958e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:00.785789 31217 ts_tablet_manager.cc:1918] T d2e6addd0c85417da8eb837fa64d18eb P 61d32e14a429452ba0597330033d3d79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:45:00.792093 31217 log.cc:1196] T d2e6addd0c85417da8eb837fa64d18eb P 61d32e14a429452ba0597330033d3d79: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-2/wal/wals/d2e6addd0c85417da8eb837fa64d18eb
I20260810 00:45:00.796991 31217 ts_tablet_manager.cc:1939] T d2e6addd0c85417da8eb837fa64d18eb P 61d32e14a429452ba0597330033d3d79: Deleting consensus metadata
I20260810 00:45:00.799789 30403 catalog_manager.cc:4841] TS 61d32e14a429452ba0597330033d3d79 (127.28.6.3:40003): tablet d2e6addd0c85417da8eb837fa64d18eb (table test-table [id=cabadf43e65c48b0a94ce21dff1a2465]) successfully deleted
I20260810 00:45:00.800462 31171 tablet_bootstrap.cc:492] T 0be5926e18d943cc90237cf719aa958e P 61d32e14a429452ba0597330033d3d79: Bootstrap starting.
I20260810 00:45:00.802327 31169 tablet_bootstrap.cc:492] T 0be5926e18d943cc90237cf719aa958e P e317b83358614dc6add668df09b881df: Bootstrap starting.
I20260810 00:45:00.803422 31171 tablet_bootstrap.cc:654] T 0be5926e18d943cc90237cf719aa958e P 61d32e14a429452ba0597330033d3d79: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:00.805976 31169 tablet_bootstrap.cc:654] T 0be5926e18d943cc90237cf719aa958e P e317b83358614dc6add668df09b881df: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:00.811647 31175 tablet_bootstrap.cc:492] T 0be5926e18d943cc90237cf719aa958e P a22ee065c94f479eb65b696fa8b44333: Bootstrap starting.
I20260810 00:45:00.817946 31175 tablet_bootstrap.cc:654] T 0be5926e18d943cc90237cf719aa958e P a22ee065c94f479eb65b696fa8b44333: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:00.829329 31220 tablet_replica.cc:330] T d2e6addd0c85417da8eb837fa64d18eb P a22ee065c94f479eb65b696fa8b44333: stopping tablet replica
I20260810 00:45:00.830041 31220 raft_consensus.cc:2229] T d2e6addd0c85417da8eb837fa64d18eb P a22ee065c94f479eb65b696fa8b44333 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:45:00.830837 31171 tablet_bootstrap.cc:492] T 0be5926e18d943cc90237cf719aa958e P 61d32e14a429452ba0597330033d3d79: No bootstrap required, opened a new log
I20260810 00:45:00.831152 31171 ts_tablet_manager.cc:1397] T 0be5926e18d943cc90237cf719aa958e P 61d32e14a429452ba0597330033d3d79: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.009s
I20260810 00:45:00.832793 31171 raft_consensus.cc:348] T 0be5926e18d943cc90237cf719aa958e P 61d32e14a429452ba0597330033d3d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22ee065c94f479eb65b696fa8b44333" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44213 } } peers { permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 } } peers { permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 } }
I20260810 00:45:00.833145 31171 raft_consensus.cc:374] T 0be5926e18d943cc90237cf719aa958e P 61d32e14a429452ba0597330033d3d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:00.833387 31171 raft_consensus.cc:729] T 0be5926e18d943cc90237cf719aa958e P 61d32e14a429452ba0597330033d3d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61d32e14a429452ba0597330033d3d79, State: Initialized, Role: FOLLOWER
I20260810 00:45:00.833987 31171 consensus_queue.cc:260] T 0be5926e18d943cc90237cf719aa958e P 61d32e14a429452ba0597330033d3d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22ee065c94f479eb65b696fa8b44333" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44213 } } peers { permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 } } peers { permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 } }
I20260810 00:45:00.835222 31175 tablet_bootstrap.cc:492] T 0be5926e18d943cc90237cf719aa958e P a22ee065c94f479eb65b696fa8b44333: No bootstrap required, opened a new log
I20260810 00:45:00.835476 31175 ts_tablet_manager.cc:1397] T 0be5926e18d943cc90237cf719aa958e P a22ee065c94f479eb65b696fa8b44333: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.007s
I20260810 00:45:00.835691 31171 ts_tablet_manager.cc:1428] T 0be5926e18d943cc90237cf719aa958e P 61d32e14a429452ba0597330033d3d79: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260810 00:45:00.836542 31218 tablet_replica.cc:330] T d2e6addd0c85417da8eb837fa64d18eb P e317b83358614dc6add668df09b881df: stopping tablet replica
I20260810 00:45:00.837114 31218 raft_consensus.cc:2229] T d2e6addd0c85417da8eb837fa64d18eb P e317b83358614dc6add668df09b881df [term 1 LEADER]: Raft consensus shutting down.
I20260810 00:45:00.838231 31218 raft_consensus.cc:2258] T d2e6addd0c85417da8eb837fa64d18eb P e317b83358614dc6add668df09b881df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:45:00.830868 31220 raft_consensus.cc:2258] T d2e6addd0c85417da8eb837fa64d18eb P a22ee065c94f479eb65b696fa8b44333 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:45:00.840864 31220 ts_tablet_manager.cc:1905] T d2e6addd0c85417da8eb837fa64d18eb P a22ee065c94f479eb65b696fa8b44333: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:45:00.843793 31169 tablet_bootstrap.cc:492] T 0be5926e18d943cc90237cf719aa958e P e317b83358614dc6add668df09b881df: No bootstrap required, opened a new log
I20260810 00:45:00.841427 31175 raft_consensus.cc:348] T 0be5926e18d943cc90237cf719aa958e P a22ee065c94f479eb65b696fa8b44333 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22ee065c94f479eb65b696fa8b44333" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44213 } } peers { permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 } } peers { permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 } }
I20260810 00:45:00.844090 31175 raft_consensus.cc:374] T 0be5926e18d943cc90237cf719aa958e P a22ee065c94f479eb65b696fa8b44333 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:00.844332 31175 raft_consensus.cc:729] T 0be5926e18d943cc90237cf719aa958e P a22ee065c94f479eb65b696fa8b44333 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a22ee065c94f479eb65b696fa8b44333, State: Initialized, Role: FOLLOWER
I20260810 00:45:00.845029 31175 consensus_queue.cc:260] T 0be5926e18d943cc90237cf719aa958e P a22ee065c94f479eb65b696fa8b44333 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22ee065c94f479eb65b696fa8b44333" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44213 } } peers { permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 } } peers { permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 } }
I20260810 00:45:00.845827 31175 ts_tablet_manager.cc:1428] T 0be5926e18d943cc90237cf719aa958e P a22ee065c94f479eb65b696fa8b44333: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260810 00:45:00.846174 31169 ts_tablet_manager.cc:1397] T 0be5926e18d943cc90237cf719aa958e P e317b83358614dc6add668df09b881df: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.015s
I20260810 00:45:00.848223 31169 raft_consensus.cc:348] T 0be5926e18d943cc90237cf719aa958e P e317b83358614dc6add668df09b881df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22ee065c94f479eb65b696fa8b44333" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44213 } } peers { permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 } } peers { permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 } }
I20260810 00:45:00.851257 31187 raft_consensus.cc:482] T 0be5926e18d943cc90237cf719aa958e P a22ee065c94f479eb65b696fa8b44333 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:45:00.851606 31169 raft_consensus.cc:374] T 0be5926e18d943cc90237cf719aa958e P e317b83358614dc6add668df09b881df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:00.851871 31169 raft_consensus.cc:729] T 0be5926e18d943cc90237cf719aa958e P e317b83358614dc6add668df09b881df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e317b83358614dc6add668df09b881df, State: Initialized, Role: FOLLOWER
I20260810 00:45:00.851573 31187 raft_consensus.cc:504] T 0be5926e18d943cc90237cf719aa958e P a22ee065c94f479eb65b696fa8b44333 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22ee065c94f479eb65b696fa8b44333" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44213 } } peers { permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 } } peers { permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 } }
I20260810 00:45:00.848618 31218 ts_tablet_manager.cc:1905] T d2e6addd0c85417da8eb837fa64d18eb P e317b83358614dc6add668df09b881df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:45:00.859288 31169 consensus_queue.cc:260] T 0be5926e18d943cc90237cf719aa958e P e317b83358614dc6add668df09b881df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22ee065c94f479eb65b696fa8b44333" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44213 } } peers { permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 } } peers { permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 } }
I20260810 00:45:00.860217 31169 ts_tablet_manager.cc:1428] T 0be5926e18d943cc90237cf719aa958e P e317b83358614dc6add668df09b881df: Time spent starting tablet: real 0.014s user 0.003s sys 0.000s
I20260810 00:45:00.861322 31187 leader_election.cc:288] T 0be5926e18d943cc90237cf719aa958e P a22ee065c94f479eb65b696fa8b44333 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e317b83358614dc6add668df09b881df (127.28.6.2:40727), 61d32e14a429452ba0597330033d3d79 (127.28.6.3:40003)
I20260810 00:45:00.873853 31218 ts_tablet_manager.cc:1918] T d2e6addd0c85417da8eb837fa64d18eb P e317b83358614dc6add668df09b881df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:45:00.874154 31218 log.cc:1196] T d2e6addd0c85417da8eb837fa64d18eb P e317b83358614dc6add668df09b881df: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-1/wal/wals/d2e6addd0c85417da8eb837fa64d18eb
I20260810 00:45:00.880426 31220 ts_tablet_manager.cc:1918] T d2e6addd0c85417da8eb837fa64d18eb P a22ee065c94f479eb65b696fa8b44333: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:45:00.880702 31220 log.cc:1196] T d2e6addd0c85417da8eb837fa64d18eb P a22ee065c94f479eb65b696fa8b44333: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-0/wal/wals/d2e6addd0c85417da8eb837fa64d18eb
I20260810 00:45:00.881469 31218 ts_tablet_manager.cc:1939] T d2e6addd0c85417da8eb837fa64d18eb P e317b83358614dc6add668df09b881df: Deleting consensus metadata
I20260810 00:45:00.883919 30404 catalog_manager.cc:4841] TS e317b83358614dc6add668df09b881df (127.28.6.2:40727): tablet d2e6addd0c85417da8eb837fa64d18eb (table test-table [id=cabadf43e65c48b0a94ce21dff1a2465]) successfully deleted
I20260810 00:45:00.884968 31220 ts_tablet_manager.cc:1939] T d2e6addd0c85417da8eb837fa64d18eb P a22ee065c94f479eb65b696fa8b44333: Deleting consensus metadata
I20260810 00:45:00.889955 30406 catalog_manager.cc:4841] TS a22ee065c94f479eb65b696fa8b44333 (127.28.6.1:44213): tablet d2e6addd0c85417da8eb837fa64d18eb (table test-table [id=cabadf43e65c48b0a94ce21dff1a2465]) successfully deleted
I20260810 00:45:00.896363 31168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:00.854979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:50453 (local address 127.28.6.2:40727)
0810 00:45:00.855143 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:00.855149 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:00.880565 (+ 25416us) server_negotiation.cc:407] Connection header received
0810 00:45:00.880778 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:00.880784 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:00.880913 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:00.881064 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:00.885920 (+ 4856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:00.887895 (+ 1975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:00.892701 (+ 4806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:00.894101 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:00.894267 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:00.894376 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:00.895720 (+ 1344us) server_negotiation.cc:299] Negotiation successful
0810 00:45:00.895898 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 00:45:00.897526 31226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:00.854565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:40727 (local address 127.28.6.1:50453)
0810 00:45:00.880451 (+ 25886us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:00.880478 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:00.880639 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:00.881317 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:00.881323 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:00.884394 (+ 3071us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:00.885580 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:00.885600 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:00.888194 (+ 2594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:00.888202 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:00.892515 (+ 4313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:00.892530 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:00.895271 (+ 2741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:00.895278 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:00.895525 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:00.895558 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 00:45:00.896761 (+ 1203us) client_negotiation.cc:240] Negotiation successful
0810 00:45:00.896900 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25664,"thread_start_us":34655,"threads_started":1}
I20260810 00:45:00.899031 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be5926e18d943cc90237cf719aa958e" candidate_uuid: "a22ee065c94f479eb65b696fa8b44333" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e317b83358614dc6add668df09b881df" is_pre_election: true
I20260810 00:45:00.899926 30871 raft_consensus.cc:2454] T 0be5926e18d943cc90237cf719aa958e P e317b83358614dc6add668df09b881df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a22ee065c94f479eb65b696fa8b44333 in term 0.
I20260810 00:45:00.901041 30553 leader_election.cc:302] T 0be5926e18d943cc90237cf719aa958e P a22ee065c94f479eb65b696fa8b44333 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a22ee065c94f479eb65b696fa8b44333, e317b83358614dc6add668df09b881df; no voters:
I20260810 00:45:00.901621 31187 raft_consensus.cc:2781] T 0be5926e18d943cc90237cf719aa958e P a22ee065c94f479eb65b696fa8b44333 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 00:45:00.901846 31187 raft_consensus.cc:482] T 0be5926e18d943cc90237cf719aa958e P a22ee065c94f479eb65b696fa8b44333 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 00:45:00.902011 31187 raft_consensus.cc:3037] T 0be5926e18d943cc90237cf719aa958e P a22ee065c94f479eb65b696fa8b44333 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:00.903770 31161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:00.856368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:50739 (local address 127.28.6.3:40003)
0810 00:45:00.856516 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:00.856522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:00.888165 (+ 31643us) server_negotiation.cc:407] Connection header received
0810 00:45:00.888605 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:00.888621 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:00.888718 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:00.888826 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:00.891550 (+ 2724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:00.893613 (+ 2063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:00.899591 (+ 5978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:00.901605 (+ 2014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:00.901725 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:00.901830 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:00.902966 (+ 1136us) server_negotiation.cc:299] Negotiation successful
0810 00:45:00.903139 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 00:45:00.908030 31225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:00.855888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:40003 (local address 127.28.6.1:50739)
0810 00:45:00.888008 (+ 32120us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:00.888041 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:00.888339 (+ 298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:00.889176 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:00.889181 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:00.889209 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:00.891262 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:00.891282 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:00.893861 (+ 2579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:00.893867 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:00.898860 (+ 4993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:00.898881 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:00.902255 (+ 3374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:00.902264 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:00.902669 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:00.902714 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 00:45:00.904051 (+ 1337us) client_negotiation.cc:240] Negotiation successful
0810 00:45:00.904205 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31821,"thread_start_us":23998,"threads_started":1}
I20260810 00:45:00.908053 31187 raft_consensus.cc:504] T 0be5926e18d943cc90237cf719aa958e P a22ee065c94f479eb65b696fa8b44333 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22ee065c94f479eb65b696fa8b44333" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44213 } } peers { permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 } } peers { permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 } }
I20260810 00:45:00.910174 31187 leader_election.cc:288] T 0be5926e18d943cc90237cf719aa958e P a22ee065c94f479eb65b696fa8b44333 [CANDIDATE]: Term 1 election: Requested vote from peers e317b83358614dc6add668df09b881df (127.28.6.2:40727), 61d32e14a429452ba0597330033d3d79 (127.28.6.3:40003)
I20260810 00:45:00.910055 31075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be5926e18d943cc90237cf719aa958e" candidate_uuid: "a22ee065c94f479eb65b696fa8b44333" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61d32e14a429452ba0597330033d3d79" is_pre_election: true
I20260810 00:45:00.910552 31075 raft_consensus.cc:2454] T 0be5926e18d943cc90237cf719aa958e P 61d32e14a429452ba0597330033d3d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a22ee065c94f479eb65b696fa8b44333 in term 0.
I20260810 00:45:00.910552 31074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be5926e18d943cc90237cf719aa958e" candidate_uuid: "a22ee065c94f479eb65b696fa8b44333" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61d32e14a429452ba0597330033d3d79"
I20260810 00:45:00.911166 31074 raft_consensus.cc:3037] T 0be5926e18d943cc90237cf719aa958e P 61d32e14a429452ba0597330033d3d79 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:00.912811 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be5926e18d943cc90237cf719aa958e" candidate_uuid: "a22ee065c94f479eb65b696fa8b44333" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e317b83358614dc6add668df09b881df"
I20260810 00:45:00.913122 30871 raft_consensus.cc:3037] T 0be5926e18d943cc90237cf719aa958e P e317b83358614dc6add668df09b881df [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:00.916749 31074 raft_consensus.cc:2454] T 0be5926e18d943cc90237cf719aa958e P 61d32e14a429452ba0597330033d3d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22ee065c94f479eb65b696fa8b44333 in term 1.
I20260810 00:45:00.917296 30550 leader_election.cc:302] T 0be5926e18d943cc90237cf719aa958e P a22ee065c94f479eb65b696fa8b44333 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61d32e14a429452ba0597330033d3d79, a22ee065c94f479eb65b696fa8b44333; no voters:
I20260810 00:45:00.917807 31187 raft_consensus.cc:2781] T 0be5926e18d943cc90237cf719aa958e P a22ee065c94f479eb65b696fa8b44333 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:45:00.919149 31187 raft_consensus.cc:686] T 0be5926e18d943cc90237cf719aa958e P a22ee065c94f479eb65b696fa8b44333 [term 1 LEADER]: Becoming Leader. State: Replica: a22ee065c94f479eb65b696fa8b44333, State: Running, Role: LEADER
I20260810 00:45:00.920113 30871 raft_consensus.cc:2454] T 0be5926e18d943cc90237cf719aa958e P e317b83358614dc6add668df09b881df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22ee065c94f479eb65b696fa8b44333 in term 1.
I20260810 00:45:00.920078 31187 consensus_queue.cc:237] T 0be5926e18d943cc90237cf719aa958e P a22ee065c94f479eb65b696fa8b44333 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22ee065c94f479eb65b696fa8b44333" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44213 } } peers { permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 } } peers { permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 } }
I20260810 00:45:00.926402 30423 catalog_manager.cc:5495] T 0be5926e18d943cc90237cf719aa958e P a22ee065c94f479eb65b696fa8b44333 reported cstate change: term changed from 0 to 1, leader changed from <none> to a22ee065c94f479eb65b696fa8b44333 (127.28.6.1). New cstate: current_term: 1 leader_uuid: "a22ee065c94f479eb65b696fa8b44333" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22ee065c94f479eb65b696fa8b44333" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 } health_report { overall_health: UNKNOWN } } }
Aug 10 00:45:01 dist-test-slave-1nl6 krb5kdc[30359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322698, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:45:01.039984 31239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:01.009739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34670 (local address 127.28.6.62:40269)
0810 00:45:01.010793 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:01.010798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:01.011702 (+ 904us) server_negotiation.cc:407] Connection header received
0810 00:45:01.012025 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:01.012028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:01.012109 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:01.012226 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:01.015505 (+ 3279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:01.016930 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:01.020998 (+ 4068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:01.021757 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:01.021864 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:01.028912 (+ 7048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:01.028921 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:01.028929 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:01.028981 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:01.035202 (+ 6221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:01.036147 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:01.036153 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:01.036157 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:01.036425 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:01.036902 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:01.036907 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:01.036911 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:01.037534 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:01.037685 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:01.039092 (+ 1407us) server_negotiation.cc:299] Negotiation successful
0810 00:45:01.039398 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":856,"thread_start_us":695,"threads_started":1}
I20260810 00:45:01.040798 31240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:01.009684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:40269 (local address 127.0.0.1:34670)
0810 00:45:01.011544 (+ 1860us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:01.011568 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:01.011875 (+ 307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:01.012355 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:01.012360 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:01.014218 (+ 1858us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:01.015253 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:01.015270 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:01.017070 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:01.017076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:01.020846 (+ 3770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:01.020862 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:01.022182 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:01.022193 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:01.022508 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:01.023478 (+ 970us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:01.023517 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:01.028501 (+ 4984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:01.035362 (+ 6861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:01.035375 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:01.035382 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:01.036006 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:01.036541 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:01.036545 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:01.036547 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:01.036769 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:01.037750 (+ 981us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:01.037758 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:01.038044 (+ 286us) client_negotiation.cc:769] Sending connection context
0810 00:45:01.038312 (+ 268us) client_negotiation.cc:240] Negotiation successful
0810 00:45:01.038652 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1580,"thread_start_us":1150,"threads_started":1}
I20260810 00:45:01.047534 30423 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:34670:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 00:45:01.048993 30423 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 00:45:01.057281 30833 tablet_service.cc:1467] Processing CreateTablet for tablet 3cb7d91057224e319ff464aca4575194 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5e444db24bff41d3be76efad1280e2be]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 00:45:01.057396 31041 tablet_service.cc:1467] Processing CreateTablet for tablet 3cb7d91057224e319ff464aca4575194 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5e444db24bff41d3be76efad1280e2be]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 00:45:01.057549 30626 tablet_service.cc:1467] Processing CreateTablet for tablet 3cb7d91057224e319ff464aca4575194 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5e444db24bff41d3be76efad1280e2be]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 00:45:01.058271 30833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cb7d91057224e319ff464aca4575194. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:01.058287 31041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cb7d91057224e319ff464aca4575194. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:01.058367 30626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cb7d91057224e319ff464aca4575194. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:01.067525 31171 tablet_bootstrap.cc:492] T 3cb7d91057224e319ff464aca4575194 P 61d32e14a429452ba0597330033d3d79: Bootstrap starting.
I20260810 00:45:01.069036 31169 tablet_bootstrap.cc:492] T 3cb7d91057224e319ff464aca4575194 P e317b83358614dc6add668df09b881df: Bootstrap starting.
I20260810 00:45:01.071278 31171 tablet_bootstrap.cc:654] T 3cb7d91057224e319ff464aca4575194 P 61d32e14a429452ba0597330033d3d79: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:01.071456 31175 tablet_bootstrap.cc:492] T 3cb7d91057224e319ff464aca4575194 P a22ee065c94f479eb65b696fa8b44333: Bootstrap starting.
I20260810 00:45:01.072428 31169 tablet_bootstrap.cc:654] T 3cb7d91057224e319ff464aca4575194 P e317b83358614dc6add668df09b881df: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:01.077107 31175 tablet_bootstrap.cc:654] T 3cb7d91057224e319ff464aca4575194 P a22ee065c94f479eb65b696fa8b44333: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:01.083026 31169 tablet_bootstrap.cc:492] T 3cb7d91057224e319ff464aca4575194 P e317b83358614dc6add668df09b881df: No bootstrap required, opened a new log
I20260810 00:45:01.083200 31169 ts_tablet_manager.cc:1397] T 3cb7d91057224e319ff464aca4575194 P e317b83358614dc6add668df09b881df: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20260810 00:45:01.084311 31169 raft_consensus.cc:348] T 3cb7d91057224e319ff464aca4575194 P e317b83358614dc6add668df09b881df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22ee065c94f479eb65b696fa8b44333" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44213 } } peers { permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 } } peers { permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 } }
I20260810 00:45:01.084631 31169 raft_consensus.cc:374] T 3cb7d91057224e319ff464aca4575194 P e317b83358614dc6add668df09b881df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:01.084741 31169 raft_consensus.cc:729] T 3cb7d91057224e319ff464aca4575194 P e317b83358614dc6add668df09b881df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e317b83358614dc6add668df09b881df, State: Initialized, Role: FOLLOWER
I20260810 00:45:01.084935 31171 tablet_bootstrap.cc:492] T 3cb7d91057224e319ff464aca4575194 P 61d32e14a429452ba0597330033d3d79: No bootstrap required, opened a new log
I20260810 00:45:01.085243 31171 ts_tablet_manager.cc:1397] T 3cb7d91057224e319ff464aca4575194 P 61d32e14a429452ba0597330033d3d79: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.012s
I20260810 00:45:01.085158 31169 consensus_queue.cc:260] T 3cb7d91057224e319ff464aca4575194 P e317b83358614dc6add668df09b881df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22ee065c94f479eb65b696fa8b44333" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44213 } } peers { permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 } } peers { permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 } }
I20260810 00:45:01.085794 31179 tablet_replica.cc:439] T 3cb7d91057224e319ff464aca4575194 P e317b83358614dc6add668df09b881df: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22ee065c94f479eb65b696fa8b44333" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44213 } } peers { permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 } } peers { permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 } } }
I20260810 00:45:01.086190 31179 tablet_replica.cc:442] T 3cb7d91057224e319ff464aca4575194 P e317b83358614dc6add668df09b881df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 00:45:01.086161 31171 raft_consensus.cc:348] T 3cb7d91057224e319ff464aca4575194 P 61d32e14a429452ba0597330033d3d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22ee065c94f479eb65b696fa8b44333" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44213 } } peers { permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 } } peers { permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 } }
I20260810 00:45:01.086488 31171 raft_consensus.cc:374] T 3cb7d91057224e319ff464aca4575194 P 61d32e14a429452ba0597330033d3d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:01.086573 31171 raft_consensus.cc:729] T 3cb7d91057224e319ff464aca4575194 P 61d32e14a429452ba0597330033d3d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61d32e14a429452ba0597330033d3d79, State: Initialized, Role: FOLLOWER
I20260810 00:45:01.086879 31171 consensus_queue.cc:260] T 3cb7d91057224e319ff464aca4575194 P 61d32e14a429452ba0597330033d3d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22ee065c94f479eb65b696fa8b44333" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44213 } } peers { permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 } } peers { permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 } }
I20260810 00:45:01.087502 31180 tablet_replica.cc:439] T 3cb7d91057224e319ff464aca4575194 P 61d32e14a429452ba0597330033d3d79: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22ee065c94f479eb65b696fa8b44333" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44213 } } peers { permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 } } peers { permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 } } }
I20260810 00:45:01.088030 31180 tablet_replica.cc:442] T 3cb7d91057224e319ff464aca4575194 P 61d32e14a429452ba0597330033d3d79: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 00:45:01.088590 31171 ts_tablet_manager.cc:1428] T 3cb7d91057224e319ff464aca4575194 P 61d32e14a429452ba0597330033d3d79: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 00:45:01.089852 31169 ts_tablet_manager.cc:1428] T 3cb7d91057224e319ff464aca4575194 P e317b83358614dc6add668df09b881df: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260810 00:45:01.090803 31175 tablet_bootstrap.cc:492] T 3cb7d91057224e319ff464aca4575194 P a22ee065c94f479eb65b696fa8b44333: No bootstrap required, opened a new log
I20260810 00:45:01.091042 31175 ts_tablet_manager.cc:1397] T 3cb7d91057224e319ff464aca4575194 P a22ee065c94f479eb65b696fa8b44333: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.009s
I20260810 00:45:01.092217 31175 raft_consensus.cc:348] T 3cb7d91057224e319ff464aca4575194 P a22ee065c94f479eb65b696fa8b44333 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22ee065c94f479eb65b696fa8b44333" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44213 } } peers { permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 } } peers { permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 } }
I20260810 00:45:01.092504 31175 raft_consensus.cc:374] T 3cb7d91057224e319ff464aca4575194 P a22ee065c94f479eb65b696fa8b44333 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:01.092593 31175 raft_consensus.cc:729] T 3cb7d91057224e319ff464aca4575194 P a22ee065c94f479eb65b696fa8b44333 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a22ee065c94f479eb65b696fa8b44333, State: Initialized, Role: FOLLOWER
I20260810 00:45:01.092864 31175 consensus_queue.cc:260] T 3cb7d91057224e319ff464aca4575194 P a22ee065c94f479eb65b696fa8b44333 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22ee065c94f479eb65b696fa8b44333" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44213 } } peers { permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 } } peers { permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 } }
I20260810 00:45:01.093410 31187 tablet_replica.cc:439] T 3cb7d91057224e319ff464aca4575194 P a22ee065c94f479eb65b696fa8b44333: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22ee065c94f479eb65b696fa8b44333" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44213 } } peers { permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 } } peers { permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 } } }
I20260810 00:45:01.093880 31187 tablet_replica.cc:442] T 3cb7d91057224e319ff464aca4575194 P a22ee065c94f479eb65b696fa8b44333: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 00:45:01.094560 31175 ts_tablet_manager.cc:1428] T 3cb7d91057224e319ff464aca4575194 P a22ee065c94f479eb65b696fa8b44333: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260810 00:45:01.118458 31187 raft_consensus.cc:482] T 3cb7d91057224e319ff464aca4575194 P a22ee065c94f479eb65b696fa8b44333 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:45:01.118781 31187 raft_consensus.cc:504] T 3cb7d91057224e319ff464aca4575194 P a22ee065c94f479eb65b696fa8b44333 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22ee065c94f479eb65b696fa8b44333" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44213 } } peers { permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 } } peers { permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 } }
I20260810 00:45:01.119673 31187 leader_election.cc:288] T 3cb7d91057224e319ff464aca4575194 P a22ee065c94f479eb65b696fa8b44333 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e317b83358614dc6add668df09b881df (127.28.6.2:40727), 61d32e14a429452ba0597330033d3d79 (127.28.6.3:40003)
I20260810 00:45:01.120302 31074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cb7d91057224e319ff464aca4575194" candidate_uuid: "a22ee065c94f479eb65b696fa8b44333" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61d32e14a429452ba0597330033d3d79" is_pre_election: true
I20260810 00:45:01.120433 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cb7d91057224e319ff464aca4575194" candidate_uuid: "a22ee065c94f479eb65b696fa8b44333" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e317b83358614dc6add668df09b881df" is_pre_election: true
I20260810 00:45:01.120651 31074 raft_consensus.cc:2454] T 3cb7d91057224e319ff464aca4575194 P 61d32e14a429452ba0597330033d3d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a22ee065c94f479eb65b696fa8b44333 in term 0.
I20260810 00:45:01.120820 30871 raft_consensus.cc:2454] T 3cb7d91057224e319ff464aca4575194 P e317b83358614dc6add668df09b881df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a22ee065c94f479eb65b696fa8b44333 in term 0.
I20260810 00:45:01.121227 30550 leader_election.cc:302] T 3cb7d91057224e319ff464aca4575194 P a22ee065c94f479eb65b696fa8b44333 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61d32e14a429452ba0597330033d3d79, a22ee065c94f479eb65b696fa8b44333; no voters:
I20260810 00:45:01.121692 31187 raft_consensus.cc:2781] T 3cb7d91057224e319ff464aca4575194 P a22ee065c94f479eb65b696fa8b44333 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 00:45:01.121897 31187 raft_consensus.cc:482] T 3cb7d91057224e319ff464aca4575194 P a22ee065c94f479eb65b696fa8b44333 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 00:45:01.122188 31187 raft_consensus.cc:3037] T 3cb7d91057224e319ff464aca4575194 P a22ee065c94f479eb65b696fa8b44333 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:01.126847 31187 raft_consensus.cc:504] T 3cb7d91057224e319ff464aca4575194 P a22ee065c94f479eb65b696fa8b44333 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22ee065c94f479eb65b696fa8b44333" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44213 } } peers { permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 } } peers { permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 } }
I20260810 00:45:01.127826 31187 leader_election.cc:288] T 3cb7d91057224e319ff464aca4575194 P a22ee065c94f479eb65b696fa8b44333 [CANDIDATE]: Term 1 election: Requested vote from peers e317b83358614dc6add668df09b881df (127.28.6.2:40727), 61d32e14a429452ba0597330033d3d79 (127.28.6.3:40003)
I20260810 00:45:01.128367 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cb7d91057224e319ff464aca4575194" candidate_uuid: "a22ee065c94f479eb65b696fa8b44333" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e317b83358614dc6add668df09b881df"
I20260810 00:45:01.128564 31074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cb7d91057224e319ff464aca4575194" candidate_uuid: "a22ee065c94f479eb65b696fa8b44333" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61d32e14a429452ba0597330033d3d79"
I20260810 00:45:01.128789 30871 raft_consensus.cc:3037] T 3cb7d91057224e319ff464aca4575194 P e317b83358614dc6add668df09b881df [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:01.128909 31074 raft_consensus.cc:3037] T 3cb7d91057224e319ff464aca4575194 P 61d32e14a429452ba0597330033d3d79 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:01.226584 31074 raft_consensus.cc:2454] T 3cb7d91057224e319ff464aca4575194 P 61d32e14a429452ba0597330033d3d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22ee065c94f479eb65b696fa8b44333 in term 1.
I20260810 00:45:01.226606 30871 raft_consensus.cc:2454] T 3cb7d91057224e319ff464aca4575194 P e317b83358614dc6add668df09b881df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22ee065c94f479eb65b696fa8b44333 in term 1.
I20260810 00:45:01.227298 30553 leader_election.cc:302] T 3cb7d91057224e319ff464aca4575194 P a22ee065c94f479eb65b696fa8b44333 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a22ee065c94f479eb65b696fa8b44333, e317b83358614dc6add668df09b881df; no voters:
I20260810 00:45:01.227902 31187 raft_consensus.cc:2781] T 3cb7d91057224e319ff464aca4575194 P a22ee065c94f479eb65b696fa8b44333 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:45:01.228256 31187 raft_consensus.cc:686] T 3cb7d91057224e319ff464aca4575194 P a22ee065c94f479eb65b696fa8b44333 [term 1 LEADER]: Becoming Leader. State: Replica: a22ee065c94f479eb65b696fa8b44333, State: Running, Role: LEADER
I20260810 00:45:01.228685 31187 consensus_queue.cc:237] T 3cb7d91057224e319ff464aca4575194 P a22ee065c94f479eb65b696fa8b44333 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22ee065c94f479eb65b696fa8b44333" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44213 } } peers { permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 } } peers { permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 } }
I20260810 00:45:01.231769 31227 tablet_replica.cc:439] T 3cb7d91057224e319ff464aca4575194 P a22ee065c94f479eb65b696fa8b44333: TxnStatusTablet state changed. Reason: New leader a22ee065c94f479eb65b696fa8b44333. Latest consensus state: current_term: 1 leader_uuid: "a22ee065c94f479eb65b696fa8b44333" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22ee065c94f479eb65b696fa8b44333" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44213 } } peers { permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 } } peers { permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 } } }
I20260810 00:45:01.232050 31227 tablet_replica.cc:442] T 3cb7d91057224e319ff464aca4575194 P a22ee065c94f479eb65b696fa8b44333: This TxnStatusTablet replica's current role is: LEADER
I20260810 00:45:01.232645 31246 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 00:45:01.232883 31246 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 00:45:01.232864 30424 catalog_manager.cc:5495] T 3cb7d91057224e319ff464aca4575194 P a22ee065c94f479eb65b696fa8b44333 reported cstate change: term changed from 0 to 1, leader changed from <none> to a22ee065c94f479eb65b696fa8b44333 (127.28.6.1). New cstate: current_term: 1 leader_uuid: "a22ee065c94f479eb65b696fa8b44333" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22ee065c94f479eb65b696fa8b44333" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 } health_report { overall_health: UNKNOWN } } }
I20260810 00:45:01.282723 31240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:01.271124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:44213 (local address 127.0.0.1:43646)
0810 00:45:01.271345 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:01.271368 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:01.271508 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:01.273292 (+ 1784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:01.273298 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:01.273315 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 00:45:01.273780 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:01.273798 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:01.275930 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:01.275936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:01.278391 (+ 2455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:01.278408 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:01.279772 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:01.279781 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:01.280131 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:01.280261 (+ 130us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 00:45:01.281890 (+ 1629us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 00:45:01.281916 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 00:45:01.282041 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 00:45:01.282205 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 00:45:01.282858 31249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:01.271526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43646 (local address 127.28.6.1:44213)
0810 00:45:01.272681 (+ 1155us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:01.272688 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:01.272715 (+ 27us) server_negotiation.cc:407] Connection header received
0810 00:45:01.272818 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:01.272824 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:01.272926 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:01.273103 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:45:01.274011 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:01.275768 (+ 1757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:01.278573 (+ 2805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:01.279489 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:01.279642 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:01.280665 (+ 1023us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:45:01.281731 (+ 1066us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:45:01.281893 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:01.282092 (+ 199us) server_negotiation.cc:299] Negotiation successful
0810 00:45:01.282185 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1003,"thread_start_us":766,"threads_started":1}
I20260810 00:45:01.291847 30871 raft_consensus.cc:1261] T 3cb7d91057224e319ff464aca4575194 P e317b83358614dc6add668df09b881df [term 1 FOLLOWER]: Refusing update from remote peer a22ee065c94f479eb65b696fa8b44333: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:45:01.293110 31227 consensus_queue.cc:1035] T 3cb7d91057224e319ff464aca4575194 P a22ee065c94f479eb65b696fa8b44333 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:45:01.298878 31074 raft_consensus.cc:1261] T 3cb7d91057224e319ff464aca4575194 P 61d32e14a429452ba0597330033d3d79 [term 1 FOLLOWER]: Refusing update from remote peer a22ee065c94f479eb65b696fa8b44333: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:45:01.299906 31227 consensus_queue.cc:1035] T 3cb7d91057224e319ff464aca4575194 P a22ee065c94f479eb65b696fa8b44333 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:45:01.307109 31227 consensus_queue.cc:1035] T 0be5926e18d943cc90237cf719aa958e P a22ee065c94f479eb65b696fa8b44333 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:45:01.318398 31180 tablet_replica.cc:439] T 3cb7d91057224e319ff464aca4575194 P 61d32e14a429452ba0597330033d3d79: TxnStatusTablet state changed. Reason: New leader a22ee065c94f479eb65b696fa8b44333. Latest consensus state: current_term: 1 leader_uuid: "a22ee065c94f479eb65b696fa8b44333" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22ee065c94f479eb65b696fa8b44333" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44213 } } peers { permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 } } peers { permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 } } }
I20260810 00:45:01.318992 31180 tablet_replica.cc:442] T 3cb7d91057224e319ff464aca4575194 P 61d32e14a429452ba0597330033d3d79: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 00:45:01.319150 31179 tablet_replica.cc:439] T 3cb7d91057224e319ff464aca4575194 P e317b83358614dc6add668df09b881df: TxnStatusTablet state changed. Reason: New leader a22ee065c94f479eb65b696fa8b44333. Latest consensus state: current_term: 1 leader_uuid: "a22ee065c94f479eb65b696fa8b44333" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22ee065c94f479eb65b696fa8b44333" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44213 } } peers { permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 } } peers { permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 } } }
I20260810 00:45:01.319669 31179 tablet_replica.cc:442] T 3cb7d91057224e319ff464aca4575194 P e317b83358614dc6add668df09b881df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 00:45:01.325445 31251 mvcc.cc:205] Tried to move back new op lower bound from 7316777784479010816 to 7316777784237645824. Current Snapshot: MvccSnapshot[applied={T|T < 7316777784479010816}]
I20260810 00:45:01.327957 31252 mvcc.cc:205] Tried to move back new op lower bound from 7316777784479010816 to 7316777784237645824. Current Snapshot: MvccSnapshot[applied={T|T < 7316777784479010816}]
I20260810 00:45:01.330876 31227 tablet_replica.cc:439] T 3cb7d91057224e319ff464aca4575194 P a22ee065c94f479eb65b696fa8b44333: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a22ee065c94f479eb65b696fa8b44333" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22ee065c94f479eb65b696fa8b44333" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44213 } } peers { permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 } } peers { permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 } } }
I20260810 00:45:01.331308 31227 tablet_replica.cc:442] T 3cb7d91057224e319ff464aca4575194 P a22ee065c94f479eb65b696fa8b44333: This TxnStatusTablet replica's current role is: LEADER
I20260810 00:45:01.332111 31187 tablet_replica.cc:439] T 3cb7d91057224e319ff464aca4575194 P a22ee065c94f479eb65b696fa8b44333: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a22ee065c94f479eb65b696fa8b44333" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22ee065c94f479eb65b696fa8b44333" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44213 } } peers { permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 } } peers { permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 } } }
I20260810 00:45:01.332873 31187 tablet_replica.cc:442] T 3cb7d91057224e319ff464aca4575194 P a22ee065c94f479eb65b696fa8b44333: This TxnStatusTablet replica's current role is: LEADER
I20260810 00:45:01.339701 31187 consensus_queue.cc:1035] T 0be5926e18d943cc90237cf719aa958e P a22ee065c94f479eb65b696fa8b44333 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:45:01.340394 31179 tablet_replica.cc:439] T 3cb7d91057224e319ff464aca4575194 P e317b83358614dc6add668df09b881df: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a22ee065c94f479eb65b696fa8b44333" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22ee065c94f479eb65b696fa8b44333" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44213 } } peers { permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 } } peers { permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 } } }
I20260810 00:45:01.340770 31179 tablet_replica.cc:442] T 3cb7d91057224e319ff464aca4575194 P e317b83358614dc6add668df09b881df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 00:45:01.340946 31180 tablet_replica.cc:439] T 3cb7d91057224e319ff464aca4575194 P 61d32e14a429452ba0597330033d3d79: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a22ee065c94f479eb65b696fa8b44333" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22ee065c94f479eb65b696fa8b44333" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44213 } } peers { permanent_uuid: "e317b83358614dc6add668df09b881df" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 40727 } } peers { permanent_uuid: "61d32e14a429452ba0597330033d3d79" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 40003 } } }
I20260810 00:45:01.341214 31180 tablet_replica.cc:442] T 3cb7d91057224e319ff464aca4575194 P 61d32e14a429452ba0597330033d3d79: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 00:45:01.378036 31249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:01.362167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43658 (local address 127.28.6.1:44213)
0810 00:45:01.362353 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:01.362366 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:01.368224 (+ 5858us) server_negotiation.cc:407] Connection header received
0810 00:45:01.368441 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:01.368460 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:01.368549 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:01.368660 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:45:01.369606 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:01.371311 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:01.374732 (+ 3421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:01.375563 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:01.375727 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:01.376884 (+ 1157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:45:01.377027 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:45:01.377137 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:01.377369 (+ 232us) server_negotiation.cc:299] Negotiation successful
0810 00:45:01.377494 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 00:45:01.439424 31249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:01.410659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:59463 (local address 127.28.6.1:44213)
0810 00:45:01.410964 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:01.410971 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:01.424396 (+ 13425us) server_negotiation.cc:407] Connection header received
0810 00:45:01.424674 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:01.424680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:01.424825 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:01.424914 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:45:01.425831 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:01.427573 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:01.436473 (+ 8900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:01.437514 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:01.437629 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:01.438423 (+ 794us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:45:01.438552 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:45:01.438611 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:01.438779 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 00:45:01.438863 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 00:45:01.440626 31274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:01.411577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:44213 (local address 127.28.6.1:59463)
0810 00:45:01.424244 (+ 12667us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:01.424266 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:01.424488 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:01.425159 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:01.425164 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:01.425176 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 00:45:01.425591 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:01.425603 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:01.427788 (+ 2185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:01.427793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:01.436316 (+ 8523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:01.436334 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:01.437826 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:01.437833 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:01.438136 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:01.438213 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 00:45:01.438668 (+ 455us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 00:45:01.438675 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 00:45:01.439669 (+ 994us) client_negotiation.cc:240] Negotiation successful
0810 00:45:01.439908 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12441,"thread_start_us":12266,"threads_started":1}
I20260810 00:45:01.577232 30424 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:45:01.577576 30424 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:45:01.581238 30424 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c467b45029c6438eb2a518a4bf45137c: Sending DeleteTablet for 3 replicas of tablet 0be5926e18d943cc90237cf719aa958e
I20260810 00:45:01.582343 30833 tablet_service.cc:1514] Processing DeleteTablet for tablet 0be5926e18d943cc90237cf719aa958e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:45:01 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:51568
I20260810 00:45:01.582840 31041 tablet_service.cc:1514] Processing DeleteTablet for tablet 0be5926e18d943cc90237cf719aa958e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:45:01 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:46906
I20260810 00:45:01.583623 30625 tablet_service.cc:1514] Processing DeleteTablet for tablet 0be5926e18d943cc90237cf719aa958e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:45:01 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:43642
I20260810 00:45:01.585196 31279 tablet_replica.cc:330] T 0be5926e18d943cc90237cf719aa958e P 61d32e14a429452ba0597330033d3d79: stopping tablet replica
I20260810 00:45:01.585691 31279 raft_consensus.cc:2229] T 0be5926e18d943cc90237cf719aa958e P 61d32e14a429452ba0597330033d3d79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:45:01.586369 31279 raft_consensus.cc:2258] T 0be5926e18d943cc90237cf719aa958e P 61d32e14a429452ba0597330033d3d79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:45:01.591665 31278 tablet_replica.cc:330] T 0be5926e18d943cc90237cf719aa958e P e317b83358614dc6add668df09b881df: stopping tablet replica
I20260810 00:45:01.591887 31278 raft_consensus.cc:2229] T 0be5926e18d943cc90237cf719aa958e P e317b83358614dc6add668df09b881df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:45:01.592362 31279 ts_tablet_manager.cc:1905] T 0be5926e18d943cc90237cf719aa958e P 61d32e14a429452ba0597330033d3d79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:45:01.592410 31278 raft_consensus.cc:2258] T 0be5926e18d943cc90237cf719aa958e P e317b83358614dc6add668df09b881df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:45:01.594521 31278 ts_tablet_manager.cc:1905] T 0be5926e18d943cc90237cf719aa958e P e317b83358614dc6add668df09b881df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:45:01.613860 31280 tablet_replica.cc:330] T 0be5926e18d943cc90237cf719aa958e P a22ee065c94f479eb65b696fa8b44333: stopping tablet replica
I20260810 00:45:01.614135 31280 raft_consensus.cc:2229] T 0be5926e18d943cc90237cf719aa958e P a22ee065c94f479eb65b696fa8b44333 [term 1 LEADER]: Raft consensus shutting down.
I20260810 00:45:01.614984 31280 raft_consensus.cc:2258] T 0be5926e18d943cc90237cf719aa958e P a22ee065c94f479eb65b696fa8b44333 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:45:01.616528 31280 ts_tablet_manager.cc:1905] T 0be5926e18d943cc90237cf719aa958e P a22ee065c94f479eb65b696fa8b44333: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 10 00:45:01 dist-test-slave-1nl6 krb5kdc[30359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322700, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.6.1@KRBTEST.COM
I20260810 00:45:01.649447 31249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:01.612571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43670 (local address 127.28.6.1:44213)
0810 00:45:01.612835 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:01.612845 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:01.613997 (+ 1152us) server_negotiation.cc:407] Connection header received
0810 00:45:01.614368 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:01.614375 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:01.614492 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:01.614606 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:01.625147 (+ 10541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:01.626721 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:01.629447 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:01.630458 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:01.630610 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:01.637626 (+ 7016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:01.637661 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:01.637684 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:01.637729 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:01.643700 (+ 5971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:01.645147 (+ 1447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:01.645158 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:01.645161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:01.645396 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:01.646041 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:01.646047 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:01.646050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:01.646707 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:01.647258 (+ 551us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:01.648756 (+ 1498us) server_negotiation.cc:299] Negotiation successful
0810 00:45:01.648989 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
W20260810 00:45:01.650954 30714 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43670
I20260810 00:45:01.691447 31287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:01.656325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34686 (local address 127.28.6.62:40269)
0810 00:45:01.671717 (+ 15392us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:01.671727 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:01.672289 (+ 562us) server_negotiation.cc:407] Connection header received
0810 00:45:01.672505 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:01.672511 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:01.672647 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:01.672777 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:01.675055 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:01.675995 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:01.677706 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:01.678442 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:01.678639 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:01.682601 (+ 3962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:01.682607 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:01.682618 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:01.682713 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:01.687556 (+ 4843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:01.688494 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:01.688501 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:01.688505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:01.688725 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:01.689276 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:01.689280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:01.689282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:01.689681 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:01.690026 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:01.690448 (+ 422us) server_negotiation.cc:299] Negotiation successful
0810 00:45:01.690859 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15196,"thread_start_us":16136,"threads_started":1}
W20260810 00:45:01.692420 30424 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34686
I20260810 00:45:01.695148 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 30528
I20260810 00:45:01.719417 31249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:01.697054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43686 (local address 127.28.6.1:44213)
0810 00:45:01.697507 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:01.697516 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:01.702711 (+ 5195us) server_negotiation.cc:407] Connection header received
0810 00:45:01.702946 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:01.702953 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:01.703029 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:01.703156 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:01.704880 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:01.705674 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:01.707334 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:01.707838 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:01.707980 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:01.711458 (+ 3478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:01.711467 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:01.711471 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:01.711503 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:01.715628 (+ 4125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:01.716429 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:01.716434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:01.716436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:01.716702 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:01.717172 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:01.717177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:01.717179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:01.717657 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:01.718090 (+ 433us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:01.718459 (+ 369us) server_negotiation.cc:299] Negotiation successful
0810 00:45:01.718778 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273}
W20260810 00:45:01.720291 30714 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43686
W20260810 00:45:01.721040 28696 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 30528: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 00:45:01.721336 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 30528
I20260810 00:45:01.740469 31279 ts_tablet_manager.cc:1918] T 0be5926e18d943cc90237cf719aa958e P 61d32e14a429452ba0597330033d3d79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 00:45:01.740434 31278 ts_tablet_manager.cc:1918] T 0be5926e18d943cc90237cf719aa958e P e317b83358614dc6add668df09b881df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 00:45:01.740732 31278 log.cc:1196] T 0be5926e18d943cc90237cf719aa958e P e317b83358614dc6add668df09b881df: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-1/wal/wals/0be5926e18d943cc90237cf719aa958e
I20260810 00:45:01.740747 31279 log.cc:1196] T 0be5926e18d943cc90237cf719aa958e P 61d32e14a429452ba0597330033d3d79: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786322687226232-28696-0/minicluster-data/ts-2/wal/wals/0be5926e18d943cc90237cf719aa958e
I20260810 00:45:01.741534 31279 ts_tablet_manager.cc:1939] T 0be5926e18d943cc90237cf719aa958e P 61d32e14a429452ba0597330033d3d79: Deleting consensus metadata
I20260810 00:45:01.741534 31278 ts_tablet_manager.cc:1939] T 0be5926e18d943cc90237cf719aa958e P e317b83358614dc6add668df09b881df: Deleting consensus metadata
I20260810 00:45:01.742890 30403 catalog_manager.cc:4841] TS 61d32e14a429452ba0597330033d3d79 (127.28.6.3:40003): tablet 0be5926e18d943cc90237cf719aa958e (table test-table [id=e318786d97b94b37b61ba08e4567c2a2]) successfully deleted
I20260810 00:45:01.742889 30404 catalog_manager.cc:4841] TS e317b83358614dc6add668df09b881df (127.28.6.2:40727): tablet 0be5926e18d943cc90237cf719aa958e (table test-table [id=e318786d97b94b37b61ba08e4567c2a2]) successfully deleted
I20260810 00:45:01.764513 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 30725
W20260810 00:45:01.764479 30406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.6.1:44213 (error 108)
I20260810 00:45:01.767025 31298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:01.765925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:44213 (local address 127.0.0.1:43688)
0810 00:45:01.766524 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:01.766742 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.1:44213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":250,"threads_started":1}
W20260810 00:45:01.767334 30406 catalog_manager.cc:4584] TS a22ee065c94f479eb65b696fa8b44333 (127.28.6.1:44213): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0be5926e18d943cc90237cf719aa958e: Network error: Client connection negotiation failed: client connection to 127.28.6.1:44213: connect: Connection refused (error 111)
Aug 10 00:45:01 dist-test-slave-1nl6 krb5kdc[30359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322700, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.6.2@KRBTEST.COM
I20260810 00:45:01.786064 31294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:01.765346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51580 (local address 127.28.6.2:40727)
0810 00:45:01.766865 (+ 1519us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:01.766872 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:01.766892 (+ 20us) server_negotiation.cc:407] Connection header received
0810 00:45:01.766979 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:01.766986 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:01.767137 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:01.767370 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:01.769031 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:01.770254 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:01.772262 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:01.772894 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:01.772994 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:01.779296 (+ 6302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:01.779309 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:01.779325 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:01.779357 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:01.782518 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:01.783419 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:01.783430 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:01.783434 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:01.783683 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:01.784070 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:01.784075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:01.784079 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:01.784717 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:01.785105 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:01.785264 (+ 159us) server_negotiation.cc:299] Negotiation successful
0810 00:45:01.785561 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1357,"thread_start_us":1130,"threads_started":1}
W20260810 00:45:01.786846 30924 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51580
W20260810 00:45:01.787691 28696 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 30725: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 00:45:01.787973 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 30725
I20260810 00:45:01.816039 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 30935
I20260810 00:45:01.820860 31298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:01.820404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:44213 (local address 127.0.0.1:43692)
0810 00:45:01.820574 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:01.820654 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.1:44213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 00:45:01.822012 31298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:01.821583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:44213 (local address 127.0.0.1:43698)
0810 00:45:01.821762 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:01.821854 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.1:44213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
Aug 10 00:45:01 dist-test-slave-1nl6 krb5kdc[30359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322700, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.6.3@KRBTEST.COM
I20260810 00:45:01.833127 31299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:01.816894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46916 (local address 127.28.6.3:40003)
0810 00:45:01.817524 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:01.817528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:01.817545 (+ 17us) server_negotiation.cc:407] Connection header received
0810 00:45:01.817600 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:01.817604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:01.817681 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:01.817772 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:01.819127 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:01.819981 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:01.821483 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:01.822130 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:01.822287 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:01.826564 (+ 4277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:01.826580 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:01.826598 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:01.826638 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:01.830254 (+ 3616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:01.830976 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:01.830984 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:01.830986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:01.831135 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:01.831530 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:01.831536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:01.831539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:01.831913 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:01.832286 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:01.832557 (+ 271us) server_negotiation.cc:299] Negotiation successful
0810 00:45:01.832799 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":341,"threads_started":1}
W20260810 00:45:01.834015 31135 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46916
W20260810 00:45:01.834869 28696 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 30935: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 00:45:01.835107 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 30935
I20260810 00:45:01.861833 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 30380
I20260810 00:45:01.863133 30485 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:45:01.981361 31298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:01.980807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:44213 (local address 127.0.0.1:43704)
0810 00:45:01.981031 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:01.981142 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.1:44213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260810 00:45:01.981400 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 30380
2026-08-10T00:45:02Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5182 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:45:02 dist-test-slave-1nl6 krb5kdc[31306](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:45:02 dist-test-slave-1nl6 krb5kdc[31306](info): set up 2 sockets
Aug 10 00:45:02 dist-test-slave-1nl6 krb5kdc[31306](info): commencing operation
krb5kdc: starting...
W20260810 00:45:04.048509 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.966s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:45:04 dist-test-slave-1nl6 krb5kdc[31306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:45:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:45:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.6.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
I20260810 00:45:04.238121 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:41269
--webserver_interface=127.28.6.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:40481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:41269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:45:04.571269 31324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:04.571570 31324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:04.571723 31324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:04.579032 31324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:45:04.579152 31324 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 00:45:04.579231 31324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:04.579317 31324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:04.579411 31324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:45:04.579496 31324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:45:04.588904 31324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:40481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:41269
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:41269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:04.590131 31324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:04.591495 31324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:04.603260 31336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:04.603323 31339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:04.603386 31335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:04.604313 31324 server_base.cc:1034] running on GCE node
I20260810 00:45:04.605103 31324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:04.606412 31324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:04.607652 31324 hybrid_clock.cc:648] HybridClock initialized: now 1786322704607597 us; error 34 us; skew 500 ppm
I20260810 00:45:04.607982 31324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:45:04 dist-test-slave-1nl6 krb5kdc[31306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322704, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:04.614704 31324 init.cc:377] Logged in from keytab as kudu/127.28.6.62@KRBTEST.COM (short username kudu)
I20260810 00:45:04.616209 31324 webserver.cc:460] Webserver started at http://127.28.6.62:34949/ using document root <none> and password file <none>
I20260810 00:45:04.616894 31324 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:04.617025 31324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:04.617367 31324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:04.620296 31324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "d7678fc95ce74a4ba1a31ad544a71001"
format_stamp: "Formatted at 2026-08-10 00:45:04 on dist-test-slave-1nl6"
server_key: "921d2afb6a6d3720cb7cd0065eb7553f"
server_key_iv: "683a540c3eb31350e1390f7cd7d7776f"
server_key_version: "encryptionkey@0"
I20260810 00:45:04.621057 31324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "d7678fc95ce74a4ba1a31ad544a71001"
format_stamp: "Formatted at 2026-08-10 00:45:04 on dist-test-slave-1nl6"
server_key: "921d2afb6a6d3720cb7cd0065eb7553f"
server_key_iv: "683a540c3eb31350e1390f7cd7d7776f"
server_key_version: "encryptionkey@0"
I20260810 00:45:04.625926 31324 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260810 00:45:04.629264 31350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:04.630438 31324 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 00:45:04.630610 31324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "d7678fc95ce74a4ba1a31ad544a71001"
format_stamp: "Formatted at 2026-08-10 00:45:04 on dist-test-slave-1nl6"
server_key: "921d2afb6a6d3720cb7cd0065eb7553f"
server_key_iv: "683a540c3eb31350e1390f7cd7d7776f"
server_key_version: "encryptionkey@0"
I20260810 00:45:04.630813 31324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:04.675077 31324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:04.679500 31324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:04.679796 31324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:04.702350 31324 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:41269
I20260810 00:45:04.702389 31437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:41269 every 8 connection(s)
I20260810 00:45:04.703791 31324 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:45:04.704592 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 31324
I20260810 00:45:04.704808 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:45:04.705633 28696 external_mini_cluster.cc:1428] Setting key b83700d140471d0ae156fa2c749d7f15
I20260810 00:45:04.708344 31438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:04.718729 31438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7678fc95ce74a4ba1a31ad544a71001: Bootstrap starting.
Aug 10 00:45:04 dist-test-slave-1nl6 krb5kdc[31306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:45:04.721987 31438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7678fc95ce74a4ba1a31ad544a71001: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:04.723147 31438 log.cc:824] T 00000000000000000000000000000000 P d7678fc95ce74a4ba1a31ad544a71001: Log is configured to *not* fsync() on all Append() calls
I20260810 00:45:04.725822 31438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7678fc95ce74a4ba1a31ad544a71001: No bootstrap required, opened a new log
I20260810 00:45:04.727478 31445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:04.707744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42158 (local address 127.28.6.62:41269)
0810 00:45:04.708587 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:04.708599 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:04.708645 (+ 46us) server_negotiation.cc:407] Connection header received
0810 00:45:04.710264 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:04.710286 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:04.710662 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:04.711094 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:04.712616 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:04.713958 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:04.715207 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:04.715875 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:04.716089 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:04.720338 (+ 4249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:04.720346 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:04.720360 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:04.720397 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:04.723474 (+ 3077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:04.724244 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:04.724251 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:04.724257 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:04.724453 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:04.724864 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:04.724868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:04.724870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:04.725338 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:04.725662 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:04.726108 (+ 446us) server_negotiation.cc:299] Negotiation successful
0810 00:45:04.726499 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":373,"threads_started":1}
I20260810 00:45:04.734427 31438 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7678fc95ce74a4ba1a31ad544a71001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7678fc95ce74a4ba1a31ad544a71001" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 41269 } }
I20260810 00:45:04.734870 31438 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7678fc95ce74a4ba1a31ad544a71001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:04.734997 31438 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7678fc95ce74a4ba1a31ad544a71001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7678fc95ce74a4ba1a31ad544a71001, State: Initialized, Role: FOLLOWER
I20260810 00:45:04.735620 31438 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7678fc95ce74a4ba1a31ad544a71001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7678fc95ce74a4ba1a31ad544a71001" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 41269 } }
I20260810 00:45:04.735930 31438 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7678fc95ce74a4ba1a31ad544a71001 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:45:04.736078 31438 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7678fc95ce74a4ba1a31ad544a71001 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:45:04.736281 31438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7678fc95ce74a4ba1a31ad544a71001 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:04.738750 31438 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7678fc95ce74a4ba1a31ad544a71001 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7678fc95ce74a4ba1a31ad544a71001" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 41269 } }
I20260810 00:45:04.739279 31438 leader_election.cc:302] T 00000000000000000000000000000000 P d7678fc95ce74a4ba1a31ad544a71001 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7678fc95ce74a4ba1a31ad544a71001; no voters:
I20260810 00:45:04.739832 31438 leader_election.cc:288] T 00000000000000000000000000000000 P d7678fc95ce74a4ba1a31ad544a71001 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:45:04.739960 31447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7678fc95ce74a4ba1a31ad544a71001 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:45:04.740676 31447 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7678fc95ce74a4ba1a31ad544a71001 [term 1 LEADER]: Becoming Leader. State: Replica: d7678fc95ce74a4ba1a31ad544a71001, State: Running, Role: LEADER
I20260810 00:45:04.741242 31447 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7678fc95ce74a4ba1a31ad544a71001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7678fc95ce74a4ba1a31ad544a71001" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 41269 } }
I20260810 00:45:04.741868 31438 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7678fc95ce74a4ba1a31ad544a71001 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:45:04.744212 31449 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7678fc95ce74a4ba1a31ad544a71001 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7678fc95ce74a4ba1a31ad544a71001. Latest consensus state: current_term: 1 leader_uuid: "d7678fc95ce74a4ba1a31ad544a71001" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7678fc95ce74a4ba1a31ad544a71001" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 41269 } } }
I20260810 00:45:04.745247 31449 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7678fc95ce74a4ba1a31ad544a71001 [sys.catalog]: This master's current role is: LEADER
I20260810 00:45:04.744740 31448 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7678fc95ce74a4ba1a31ad544a71001 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7678fc95ce74a4ba1a31ad544a71001" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7678fc95ce74a4ba1a31ad544a71001" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 41269 } } }
I20260810 00:45:04.745798 31448 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7678fc95ce74a4ba1a31ad544a71001 [sys.catalog]: This master's current role is: LEADER
I20260810 00:45:04.748009 31455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:45:04.754575 31455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:45:04.764986 31455 catalog_manager.cc:1347] Generated new cluster ID: 16d4804046264e7f8959ca9b50f5323f
I20260810 00:45:04.765205 31455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:45:04.778690 31455 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:45:04.779933 31455 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:45:04.791610 31455 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7678fc95ce74a4ba1a31ad544a71001: Generated new TSK 0
I20260810 00:45:04.792791 31455 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.6.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab.
Entry for principal kudu/127.28.6.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab.
I20260810 00:45:04.853046 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.1:0
--local_ip_for_outbound_sockets=127.28.6.1
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:41269
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:40481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:45:05.180675 31476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:05.180963 31476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:05.181133 31476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:05.188309 31476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:05.188467 31476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:05.188608 31476 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 00:45:05.188692 31476 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 00:45:05.188800 31476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.1
I20260810 00:45:05.198088 31476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:40481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.6.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.6.62:41269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.1
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:05.199301 31476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:05.200590 31476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:05.213191 31484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:05.213217 31482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:05.213567 31489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:05.214257 31476 server_base.cc:1034] running on GCE node
I20260810 00:45:05.214880 31476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:05.215809 31476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:05.217057 31476 hybrid_clock.cc:648] HybridClock initialized: now 1786322705217006 us; error 64 us; skew 500 ppm
I20260810 00:45:05.217382 31476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:45:05 dist-test-slave-1nl6 krb5kdc[31306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322705, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:05.220539 31476 init.cc:377] Logged in from keytab as kudu/127.28.6.1@KRBTEST.COM (short username kudu)
I20260810 00:45:05.221911 31476 webserver.cc:460] Webserver started at http://127.28.6.1:45223/ using document root <none> and password file <none>
I20260810 00:45:05.222517 31476 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:05.222679 31476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:05.223080 31476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:05.225859 31476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-0/data/instance:
uuid: "875b3956c2304204aff06dbd87048d09"
format_stamp: "Formatted at 2026-08-10 00:45:05 on dist-test-slave-1nl6"
server_key: "6e72b71bbc7d0b906c7e530b80c79d06"
server_key_iv: "915f3fea67cb75f00fe3854805a0e864"
server_key_version: "encryptionkey@0"
I20260810 00:45:05.226601 31476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance:
uuid: "875b3956c2304204aff06dbd87048d09"
format_stamp: "Formatted at 2026-08-10 00:45:05 on dist-test-slave-1nl6"
server_key: "6e72b71bbc7d0b906c7e530b80c79d06"
server_key_iv: "915f3fea67cb75f00fe3854805a0e864"
server_key_version: "encryptionkey@0"
I20260810 00:45:05.231540 31476 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260810 00:45:05.235172 31498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:05.236395 31476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 00:45:05.236564 31476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-0/wal
uuid: "875b3956c2304204aff06dbd87048d09"
format_stamp: "Formatted at 2026-08-10 00:45:05 on dist-test-slave-1nl6"
server_key: "6e72b71bbc7d0b906c7e530b80c79d06"
server_key_iv: "915f3fea67cb75f00fe3854805a0e864"
server_key_version: "encryptionkey@0"
I20260810 00:45:05.236757 31476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:05.260695 31476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:05.264987 31476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:05.265334 31476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:05.267987 31476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:45:05.268082 31476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:05.268245 31476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:45:05.268360 31476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 00:45:05 dist-test-slave-1nl6 krb5kdc[31306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322705, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:45:05.292917 31511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:05.268412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:35235 (local address 127.28.6.62:41269)
0810 00:45:05.268902 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:05.268909 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:05.270157 (+ 1248us) server_negotiation.cc:407] Connection header received
0810 00:45:05.272458 (+ 2301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:05.272468 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:05.272638 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:05.272792 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:05.275668 (+ 2876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:05.276729 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:05.279454 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:05.279910 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:05.280063 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:05.285237 (+ 5174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:05.285248 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:05.285256 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:05.285312 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:05.289078 (+ 3766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:05.290118 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:05.290125 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:05.290129 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:05.290289 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:05.290826 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:05.290832 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:05.290835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:05.291187 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:05.291356 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:05.292323 (+ 967us) server_negotiation.cc:299] Negotiation successful
0810 00:45:05.292540 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":253,"threads_started":1}
I20260810 00:45:05.293871 31515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:05.269067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:41269 (local address 127.28.6.1:35235)
0810 00:45:05.269970 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:05.270026 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:05.272054 (+ 2028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:05.273384 (+ 1330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:05.273399 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:05.274290 (+ 891us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:05.275419 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:05.275456 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:05.276878 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:05.276886 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:05.279293 (+ 2407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:05.279309 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:05.280177 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:05.280190 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:05.280556 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:05.281349 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:05.281390 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:05.284902 (+ 3512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:05.289237 (+ 4335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:05.289245 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:05.289265 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:05.289917 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:05.290417 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:05.290421 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:05.290425 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:05.290679 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:05.291330 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:05.291338 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:05.291836 (+ 498us) client_negotiation.cc:769] Sending connection context
0810 00:45:05.292222 (+ 386us) client_negotiation.cc:240] Negotiation successful
0810 00:45:05.292725 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":383,"threads_started":1}
I20260810 00:45:05.314503 31595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:05.299754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:41269 (local address 127.28.6.1:40635)
0810 00:45:05.300415 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:05.300439 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:05.300542 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:05.300930 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:05.300932 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:05.301335 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:05.301593 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:05.301601 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:05.303249 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:05.303257 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:05.304832 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:05.304854 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:05.306005 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:05.306017 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:05.306321 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:05.307051 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:05.307079 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:05.308080 (+ 1001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:05.311653 (+ 3573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:05.311663 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:05.311667 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:05.312282 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:05.312824 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:05.312827 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:05.312829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:05.312996 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:05.313656 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:05.313660 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:05.313816 (+ 156us) client_negotiation.cc:769] Sending connection context
0810 00:45:05.313950 (+ 134us) client_negotiation.cc:240] Negotiation successful
0810 00:45:05.314180 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":373,"threads_started":1}
I20260810 00:45:05.314759 31511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:05.299635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:40635 (local address 127.28.6.62:41269)
0810 00:45:05.299850 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:05.299859 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:05.300523 (+ 664us) server_negotiation.cc:407] Connection header received
0810 00:45:05.300663 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:05.300672 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:05.300782 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:05.300916 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:05.301726 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:05.303088 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:05.305007 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:05.305787 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:05.305943 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:05.308273 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:05.308279 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:05.308284 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:05.308321 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:05.311516 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:05.312480 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:05.312488 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:05.312492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:05.312663 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:05.313144 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:05.313150 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:05.313156 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:05.313539 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:05.313696 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:05.314101 (+ 405us) server_negotiation.cc:299] Negotiation successful
0810 00:45:05.314339 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 00:45:05.322778 31476 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.1:45757
I20260810 00:45:05.322821 31678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.1:45757 every 8 connection(s)
I20260810 00:45:05.323982 31476 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
I20260810 00:45:05.325969 31679 heartbeater.cc:347] Connected to a master server at 127.28.6.62:41269
I20260810 00:45:05.326253 31679 heartbeater.cc:464] Registering TS with master...
I20260810 00:45:05.326370 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 31476
I20260810 00:45:05.326572 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance
I20260810 00:45:05.327100 31679 heartbeater.cc:511] Master 127.28.6.62:41269 requested a full tablet report, sending...
I20260810 00:45:05.327487 28696 external_mini_cluster.cc:1428] Setting key 44589d31965721ba46547921aaedb72c
I20260810 00:45:05.329222 31372 ts_manager.cc:194] Registered new tserver with Master: 875b3956c2304204aff06dbd87048d09 (127.28.6.1:45757)
I20260810 00:45:05.330636 31372 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.1@KRBTEST.COM'} at 127.28.6.1:35235
WARNING: no policy specified for kudu/127.28.6.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab.
Entry for principal kudu/127.28.6.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab.
I20260810 00:45:05.370092 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.2:0
--local_ip_for_outbound_sockets=127.28.6.2
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:41269
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:40481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:45:05.714326 31686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:05.714592 31686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:05.714763 31686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:05.722003 31686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:05.722121 31686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:05.722277 31686 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 00:45:05.722359 31686 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 00:45:05.722494 31686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.2
I20260810 00:45:05.732175 31686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:40481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.6.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.6.62:41269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.2
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:05.733393 31686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:05.735038 31686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:05.749810 31697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:05.749903 31694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:05.750029 31692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:05.750833 31686 server_base.cc:1034] running on GCE node
I20260810 00:45:05.751506 31686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:05.752378 31686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:05.753583 31686 hybrid_clock.cc:648] HybridClock initialized: now 1786322705753546 us; error 26 us; skew 500 ppm
I20260810 00:45:05.753916 31686 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:45:05 dist-test-slave-1nl6 krb5kdc[31306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322705, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:05.757236 31686 init.cc:377] Logged in from keytab as kudu/127.28.6.2@KRBTEST.COM (short username kudu)
I20260810 00:45:05.758569 31686 webserver.cc:460] Webserver started at http://127.28.6.2:46489/ using document root <none> and password file <none>
I20260810 00:45:05.759267 31686 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:05.759375 31686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:05.759637 31686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:05.762476 31686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-1/data/instance:
uuid: "433319dcedfe4887857a97939ba14401"
format_stamp: "Formatted at 2026-08-10 00:45:05 on dist-test-slave-1nl6"
server_key: "d322fce211aedc1bfc8094ed2bbf2e69"
server_key_iv: "b29b5d8e13228974bedbf0c364b39708"
server_key_version: "encryptionkey@0"
I20260810 00:45:05.763243 31686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance:
uuid: "433319dcedfe4887857a97939ba14401"
format_stamp: "Formatted at 2026-08-10 00:45:05 on dist-test-slave-1nl6"
server_key: "d322fce211aedc1bfc8094ed2bbf2e69"
server_key_iv: "b29b5d8e13228974bedbf0c364b39708"
server_key_version: "encryptionkey@0"
I20260810 00:45:05.768249 31686 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260810 00:45:05.771561 31711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:05.772812 31686 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 00:45:05.772992 31686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-1/wal
uuid: "433319dcedfe4887857a97939ba14401"
format_stamp: "Formatted at 2026-08-10 00:45:05 on dist-test-slave-1nl6"
server_key: "d322fce211aedc1bfc8094ed2bbf2e69"
server_key_iv: "b29b5d8e13228974bedbf0c364b39708"
server_key_version: "encryptionkey@0"
I20260810 00:45:05.773185 31686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:05.796988 31686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:05.801236 31686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:05.801548 31686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:05.804352 31686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:45:05.804488 31686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:05.804642 31686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:45:05.804783 31686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Aug 10 00:45:05 dist-test-slave-1nl6 krb5kdc[31306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322705, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:45:05.826545 31511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:05.804573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:43419 (local address 127.28.6.62:41269)
0810 00:45:05.804795 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:05.804804 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:05.806336 (+ 1532us) server_negotiation.cc:407] Connection header received
0810 00:45:05.809261 (+ 2925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:05.809265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:05.809362 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:05.809496 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:05.812101 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:05.813050 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:05.815636 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:05.816047 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:05.816183 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:05.820478 (+ 4295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:05.820483 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:05.820486 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:05.820519 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:05.823121 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:05.824103 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:05.824111 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:05.824116 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:05.824281 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:05.824869 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:05.824873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:05.824875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:05.825146 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:05.825308 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:05.825983 (+ 675us) server_negotiation.cc:299] Negotiation successful
0810 00:45:05.826191 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 00:45:05.827746 31724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:05.805105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:41269 (local address 127.28.6.2:43419)
0810 00:45:05.806105 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:05.806174 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:05.808938 (+ 2764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:05.809825 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:05.809841 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:05.810777 (+ 936us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:05.811872 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:05.811904 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:05.813224 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:05.813233 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:05.815480 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:05.815496 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:05.816280 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:05.816290 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:05.816510 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:05.817142 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:05.817178 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:05.820223 (+ 3045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:05.823260 (+ 3037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:05.823269 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:05.823288 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:05.823927 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:05.824430 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:05.824438 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:05.824448 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:05.824720 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:05.825285 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:05.825292 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:05.825632 (+ 340us) client_negotiation.cc:769] Sending connection context
0810 00:45:05.825884 (+ 252us) client_negotiation.cc:240] Negotiation successful
0810 00:45:05.826353 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":397,"threads_started":1}
I20260810 00:45:05.847667 31511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:05.833019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:33591 (local address 127.28.6.62:41269)
0810 00:45:05.833183 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:05.833187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:05.833958 (+ 771us) server_negotiation.cc:407] Connection header received
0810 00:45:05.834150 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:05.834156 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:05.834231 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:05.834337 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:05.835671 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:05.836376 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:05.837802 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:05.838256 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:05.838399 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:05.841372 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:05.841378 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:05.841382 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:05.841411 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:05.844733 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:05.845679 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:05.845684 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:05.845689 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:05.845826 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:05.846205 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:05.846210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:05.846213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:05.846519 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:05.846668 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:05.847104 (+ 436us) server_negotiation.cc:299] Negotiation successful
0810 00:45:05.847333 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 00:45:05.847779 31778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:05.833149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:41269 (local address 127.28.6.2:33591)
0810 00:45:05.833862 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:05.833878 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:05.834023 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:05.834364 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:05.834369 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:05.835036 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:05.835524 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:05.835538 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:05.836634 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:05.836640 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:05.837651 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:05.837662 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:05.838529 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:05.838540 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:05.838847 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:05.839591 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:05.839619 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:05.841095 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:05.844869 (+ 3774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:05.844874 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:05.844882 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:05.845552 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:05.845940 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:05.845945 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:05.845949 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:05.846077 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:05.846637 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:05.846642 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:05.846825 (+ 183us) client_negotiation.cc:769] Sending connection context
0810 00:45:05.847007 (+ 182us) client_negotiation.cc:240] Negotiation successful
0810 00:45:05.847324 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":365,"threads_started":1}
I20260810 00:45:05.863715 31686 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.2:46687
I20260810 00:45:05.863798 31879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.2:46687 every 8 connection(s)
I20260810 00:45:05.864961 31686 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
I20260810 00:45:05.866984 31880 heartbeater.cc:347] Connected to a master server at 127.28.6.62:41269
I20260810 00:45:05.867173 31880 heartbeater.cc:464] Registering TS with master...
I20260810 00:45:05.867744 31880 heartbeater.cc:511] Master 127.28.6.62:41269 requested a full tablet report, sending...
I20260810 00:45:05.869096 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 31686
I20260810 00:45:05.869139 31372 ts_manager.cc:194] Registered new tserver with Master: 433319dcedfe4887857a97939ba14401 (127.28.6.2:46687)
I20260810 00:45:05.869287 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance
I20260810 00:45:05.869844 28696 external_mini_cluster.cc:1428] Setting key f908d6c83b84f631d6aabec701950443
I20260810 00:45:05.870253 31372 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.2@KRBTEST.COM'} at 127.28.6.2:43419
WARNING: no policy specified for kudu/127.28.6.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab.
Entry for principal kudu/127.28.6.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab.
I20260810 00:45:05.912473 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.3:0
--local_ip_for_outbound_sockets=127.28.6.3
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:41269
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:40481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:45:06.237506 31884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:06.237761 31884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:06.237946 31884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:06.245121 31884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:06.245239 31884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:06.245400 31884 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 00:45:06.245479 31884 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 00:45:06.245613 31884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.3
I20260810 00:45:06.255164 31884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:40481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.6.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.6.62:41269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.3
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:06.256402 31884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:06.257692 31884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:06.270995 31891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:06.271030 31892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:06.271349 31894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:06.271862 31884 server_base.cc:1034] running on GCE node
I20260810 00:45:06.272480 31884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:06.273356 31884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:06.274580 31884 hybrid_clock.cc:648] HybridClock initialized: now 1786322706274537 us; error 52 us; skew 500 ppm
I20260810 00:45:06.274891 31884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:45:06 dist-test-slave-1nl6 krb5kdc[31306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322706, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:06.278010 31884 init.cc:377] Logged in from keytab as kudu/127.28.6.3@KRBTEST.COM (short username kudu)
I20260810 00:45:06.279414 31884 webserver.cc:460] Webserver started at http://127.28.6.3:41473/ using document root <none> and password file <none>
I20260810 00:45:06.280015 31884 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:06.280148 31884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:06.280478 31884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:06.283476 31884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-2/data/instance:
uuid: "658b70fbcba84147b8c6bb3dee887e7d"
format_stamp: "Formatted at 2026-08-10 00:45:06 on dist-test-slave-1nl6"
server_key: "07b190dfbf69125f9e1e93572fd019fd"
server_key_iv: "12232a24fc4809a7492ee0f12e06fb92"
server_key_version: "encryptionkey@0"
I20260810 00:45:06.284235 31884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance:
uuid: "658b70fbcba84147b8c6bb3dee887e7d"
format_stamp: "Formatted at 2026-08-10 00:45:06 on dist-test-slave-1nl6"
server_key: "07b190dfbf69125f9e1e93572fd019fd"
server_key_iv: "12232a24fc4809a7492ee0f12e06fb92"
server_key_version: "encryptionkey@0"
I20260810 00:45:06.289233 31884 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260810 00:45:06.292524 31906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:06.293721 31884 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 00:45:06.293900 31884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-2/wal
uuid: "658b70fbcba84147b8c6bb3dee887e7d"
format_stamp: "Formatted at 2026-08-10 00:45:06 on dist-test-slave-1nl6"
server_key: "07b190dfbf69125f9e1e93572fd019fd"
server_key_iv: "12232a24fc4809a7492ee0f12e06fb92"
server_key_version: "encryptionkey@0"
I20260810 00:45:06.294098 31884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:06.319083 31884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:06.322788 31884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:06.323117 31884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:06.325704 31884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:45:06.325845 31884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:06.325990 31884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:45:06.326112 31884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:06.334012 31679 heartbeater.cc:503] Master 127.28.6.62:41269 was elected leader, sending a full tablet report...
Aug 10 00:45:06 dist-test-slave-1nl6 krb5kdc[31306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322706, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:45:06.350942 31511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:06.326547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:49595 (local address 127.28.6.62:41269)
0810 00:45:06.326743 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:06.326750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:06.328279 (+ 1529us) server_negotiation.cc:407] Connection header received
0810 00:45:06.330357 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:06.330366 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:06.330466 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:06.330599 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:06.332626 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.333970 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.337425 (+ 3455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.337925 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.338026 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:06.343504 (+ 5478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:06.343514 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:06.343517 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:06.343550 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:06.346428 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:06.347290 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:06.347299 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:06.347303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:06.347488 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:06.348091 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:06.348096 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:06.348101 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:06.348725 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:06.348890 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:06.350051 (+ 1161us) server_negotiation.cc:299] Negotiation successful
0810 00:45:06.350388 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 00:45:06.351541 31919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:06.327108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:41269 (local address 127.28.6.3:49595)
0810 00:45:06.328097 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:06.328145 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:06.330057 (+ 1912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:06.330950 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:06.330963 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:06.331496 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:06.332322 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:06.332345 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.334205 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.334214 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:06.337246 (+ 3032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:06.337263 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.338180 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.338191 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:06.338596 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:06.339469 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:06.339491 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:06.343169 (+ 3678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:06.346580 (+ 3411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:06.346589 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:06.346604 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:06.347143 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:06.347650 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:06.347657 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:06.347664 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:06.347904 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:06.348895 (+ 991us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:06.348904 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:06.349460 (+ 556us) client_negotiation.cc:769] Sending connection context
0810 00:45:06.349834 (+ 374us) client_negotiation.cc:240] Negotiation successful
0810 00:45:06.350292 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":361,"threads_started":1}
I20260810 00:45:06.382359 31511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:06.363481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:45569 (local address 127.28.6.62:41269)
0810 00:45:06.363767 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:06.363776 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:06.367404 (+ 3628us) server_negotiation.cc:407] Connection header received
0810 00:45:06.367624 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:06.367637 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:06.367696 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:06.367799 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:06.369315 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.370458 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.372874 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.373437 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.373579 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:06.376185 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:06.376197 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:06.376201 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:06.376231 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:06.379193 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:06.380245 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:06.380254 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:06.380260 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:06.380411 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:06.380814 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:06.380818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:06.380822 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:06.381157 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:06.381334 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:06.381760 (+ 426us) server_negotiation.cc:299] Negotiation successful
0810 00:45:06.381992 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260810 00:45:06.382459 32006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:06.362840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:41269 (local address 127.28.6.3:45569)
0810 00:45:06.367287 (+ 4447us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:06.367303 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:06.367487 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:06.367839 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:06.367844 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:06.368577 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:06.369159 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:06.369173 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.370648 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.370654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:06.372466 (+ 1812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:06.372708 (+ 242us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.373852 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.373863 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:06.374058 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:06.374722 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:06.374763 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:06.375981 (+ 1218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:06.379402 (+ 3421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:06.379410 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:06.379417 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:06.380089 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:06.380557 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:06.380562 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:06.380565 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:06.380702 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:06.381282 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:06.381286 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:06.381483 (+ 197us) client_negotiation.cc:769] Sending connection context
0810 00:45:06.381612 (+ 129us) client_negotiation.cc:240] Negotiation successful
0810 00:45:06.381975 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4305,"thread_start_us":4040,"threads_started":1}
I20260810 00:45:06.391456 31884 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.3:41205
I20260810 00:45:06.391537 32077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.3:41205 every 8 connection(s)
I20260810 00:45:06.392573 31884 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
I20260810 00:45:06.394467 32078 heartbeater.cc:347] Connected to a master server at 127.28.6.62:41269
I20260810 00:45:06.394644 32078 heartbeater.cc:464] Registering TS with master...
I20260810 00:45:06.395341 32078 heartbeater.cc:511] Master 127.28.6.62:41269 requested a full tablet report, sending...
I20260810 00:45:06.396627 31372 ts_manager.cc:194] Registered new tserver with Master: 658b70fbcba84147b8c6bb3dee887e7d (127.28.6.3:41205)
I20260810 00:45:06.397509 31372 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.3@KRBTEST.COM'} at 127.28.6.3:49595
I20260810 00:45:06.399434 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 31884
I20260810 00:45:06.399616 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance
I20260810 00:45:06.400137 28696 external_mini_cluster.cc:1428] Setting key 2d9bbaf595433875b434b97d05fa33d7
I20260810 00:45:06.403056 28696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 00:45:06 dist-test-slave-1nl6 krb5kdc[31306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 00:45:06 dist-test-slave-1nl6 krb5kdc[31306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:45:06.452353 31511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:06.438534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42160 (local address 127.28.6.62:41269)
0810 00:45:06.438761 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:06.438768 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:06.439023 (+ 255us) server_negotiation.cc:407] Connection header received
0810 00:45:06.439180 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:06.439184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:06.439250 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:06.439350 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:06.440703 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.441516 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.442887 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.443396 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.443492 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:06.447424 (+ 3932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:06.447429 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:06.447433 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:06.447462 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:06.450019 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:06.450588 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:06.450591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:06.450594 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:06.450715 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:06.451097 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:06.451101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:06.451103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:06.451359 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:06.451541 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:06.451774 (+ 233us) server_negotiation.cc:299] Negotiation successful
0810 00:45:06.451940 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 00:45:06.456696 31372 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42160:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 00:45:06.463593 31372 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 00:45:06.493080 32099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:06.475262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:46687 (local address 127.0.0.1:48322)
0810 00:45:06.476301 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:06.476348 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:06.476572 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:06.482499 (+ 5927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:06.482510 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:06.482563 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:06.483206 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:06.483323 (+ 117us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.486847 (+ 3524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.486861 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:06.489953 (+ 3092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:06.489969 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.491918 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.491924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:06.492123 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:06.492202 (+ 79us) client_negotiation.cc:769] Sending connection context
0810 00:45:06.492331 (+ 129us) client_negotiation.cc:240] Negotiation successful
0810 00:45:06.492553 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":28,"thread_start_us":743,"threads_started":1}
I20260810 00:45:06.493631 32101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:06.475532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48322 (local address 127.28.6.2:46687)
0810 00:45:06.481753 (+ 6221us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:06.481761 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:06.481790 (+ 29us) server_negotiation.cc:407] Connection header received
0810 00:45:06.481903 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:06.481913 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:06.482191 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:06.483184 (+ 993us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:06.484740 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.486625 (+ 1885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.490217 (+ 3592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.491673 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.491889 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:06.492099 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:06.492929 (+ 830us) server_negotiation.cc:299] Negotiation successful
0810 00:45:06.493121 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6078,"thread_start_us":5956,"threads_started":1}
I20260810 00:45:06.493819 32105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:06.475813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39678 (local address 127.28.6.1:45757)
0810 00:45:06.481319 (+ 5506us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:06.481334 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:06.481812 (+ 478us) server_negotiation.cc:407] Connection header received
0810 00:45:06.482164 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:06.482175 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:06.482479 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:06.482679 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:06.484261 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.486325 (+ 2064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.490953 (+ 4628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.492448 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.492577 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:06.492710 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:06.493362 (+ 652us) server_negotiation.cc:299] Negotiation successful
0810 00:45:06.493525 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5292,"thread_start_us":5134,"threads_started":1}
I20260810 00:45:06.494582 32104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:06.475617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:45757 (local address 127.0.0.1:39678)
0810 00:45:06.481707 (+ 6090us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:06.481725 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:06.481923 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:06.483025 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:06.483032 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:06.483072 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:06.484019 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:06.484034 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.486561 (+ 2527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.486569 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:06.490706 (+ 4137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:06.490726 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.492880 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.492887 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:06.493110 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:06.493147 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 00:45:06.494090 (+ 943us) client_negotiation.cc:240] Negotiation successful
0810 00:45:06.494235 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5955,"mutex_wait_us":788,"thread_start_us":5786,"threads_started":1}
I20260810 00:45:06.497529 31787 tablet_service.cc:1467] Processing CreateTablet for tablet 1ca64df3ab584cdea699ba758e88a0f7 (DEFAULT_TABLE table=test-table [id=2213e988d3f94fd88ec9c34dd970eda1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:06.497932 31575 tablet_service.cc:1467] Processing CreateTablet for tablet 1ca64df3ab584cdea699ba758e88a0f7 (DEFAULT_TABLE table=test-table [id=2213e988d3f94fd88ec9c34dd970eda1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:06.499312 31575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ca64df3ab584cdea699ba758e88a0f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:06.499334 31787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ca64df3ab584cdea699ba758e88a0f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:06.499821 32108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:06.481794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:41205 (local address 127.0.0.1:36782)
0810 00:45:06.491487 (+ 9693us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:06.491505 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:06.491633 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:06.492246 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:06.492250 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:06.492278 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:06.492650 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:06.492664 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.494663 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.494671 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:06.496795 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:06.496811 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.499046 (+ 2235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.499053 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:06.499220 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:06.499253 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 00:45:06.499387 (+ 134us) client_negotiation.cc:240] Negotiation successful
0810 00:45:06.499517 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9578,"thread_start_us":9453,"threads_started":1}
I20260810 00:45:06.499958 32109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:06.482073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36782 (local address 127.28.6.3:41205)
0810 00:45:06.490382 (+ 8309us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:06.490394 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:06.491618 (+ 1224us) server_negotiation.cc:407] Connection header received
0810 00:45:06.491795 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:06.491802 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:06.492040 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:06.492250 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:06.492897 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.494486 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.496983 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.498518 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.498653 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:06.498887 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:06.499453 (+ 566us) server_negotiation.cc:299] Negotiation successful
0810 00:45:06.499622 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8065,"thread_start_us":7823,"threads_started":1}
I20260810 00:45:06.503667 31975 tablet_service.cc:1467] Processing CreateTablet for tablet 1ca64df3ab584cdea699ba758e88a0f7 (DEFAULT_TABLE table=test-table [id=2213e988d3f94fd88ec9c34dd970eda1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:06.505122 31975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ca64df3ab584cdea699ba758e88a0f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:06.511073 32110 tablet_bootstrap.cc:492] T 1ca64df3ab584cdea699ba758e88a0f7 P 875b3956c2304204aff06dbd87048d09: Bootstrap starting.
I20260810 00:45:06.513875 32111 tablet_bootstrap.cc:492] T 1ca64df3ab584cdea699ba758e88a0f7 P 433319dcedfe4887857a97939ba14401: Bootstrap starting.
I20260810 00:45:06.514497 32110 tablet_bootstrap.cc:654] T 1ca64df3ab584cdea699ba758e88a0f7 P 875b3956c2304204aff06dbd87048d09: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:06.515959 32110 log.cc:824] T 1ca64df3ab584cdea699ba758e88a0f7 P 875b3956c2304204aff06dbd87048d09: Log is configured to *not* fsync() on all Append() calls
I20260810 00:45:06.517570 32111 tablet_bootstrap.cc:654] T 1ca64df3ab584cdea699ba758e88a0f7 P 433319dcedfe4887857a97939ba14401: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:06.519234 32111 log.cc:824] T 1ca64df3ab584cdea699ba758e88a0f7 P 433319dcedfe4887857a97939ba14401: Log is configured to *not* fsync() on all Append() calls
I20260810 00:45:06.519639 32114 tablet_bootstrap.cc:492] T 1ca64df3ab584cdea699ba758e88a0f7 P 658b70fbcba84147b8c6bb3dee887e7d: Bootstrap starting.
I20260810 00:45:06.522277 32114 tablet_bootstrap.cc:654] T 1ca64df3ab584cdea699ba758e88a0f7 P 658b70fbcba84147b8c6bb3dee887e7d: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:06.523561 32114 log.cc:824] T 1ca64df3ab584cdea699ba758e88a0f7 P 658b70fbcba84147b8c6bb3dee887e7d: Log is configured to *not* fsync() on all Append() calls
I20260810 00:45:06.525090 32110 tablet_bootstrap.cc:492] T 1ca64df3ab584cdea699ba758e88a0f7 P 875b3956c2304204aff06dbd87048d09: No bootstrap required, opened a new log
I20260810 00:45:06.525118 32111 tablet_bootstrap.cc:492] T 1ca64df3ab584cdea699ba758e88a0f7 P 433319dcedfe4887857a97939ba14401: No bootstrap required, opened a new log
I20260810 00:45:06.525367 32110 ts_tablet_manager.cc:1397] T 1ca64df3ab584cdea699ba758e88a0f7 P 875b3956c2304204aff06dbd87048d09: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20260810 00:45:06.525389 32111 ts_tablet_manager.cc:1397] T 1ca64df3ab584cdea699ba758e88a0f7 P 433319dcedfe4887857a97939ba14401: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260810 00:45:06.526499 32114 tablet_bootstrap.cc:492] T 1ca64df3ab584cdea699ba758e88a0f7 P 658b70fbcba84147b8c6bb3dee887e7d: No bootstrap required, opened a new log
I20260810 00:45:06.526760 32114 ts_tablet_manager.cc:1397] T 1ca64df3ab584cdea699ba758e88a0f7 P 658b70fbcba84147b8c6bb3dee887e7d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 00:45:06.533141 32110 raft_consensus.cc:348] T 1ca64df3ab584cdea699ba758e88a0f7 P 875b3956c2304204aff06dbd87048d09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:06.533521 32110 raft_consensus.cc:374] T 1ca64df3ab584cdea699ba758e88a0f7 P 875b3956c2304204aff06dbd87048d09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:06.533654 32110 raft_consensus.cc:729] T 1ca64df3ab584cdea699ba758e88a0f7 P 875b3956c2304204aff06dbd87048d09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 875b3956c2304204aff06dbd87048d09, State: Initialized, Role: FOLLOWER
I20260810 00:45:06.534318 32110 consensus_queue.cc:260] T 1ca64df3ab584cdea699ba758e88a0f7 P 875b3956c2304204aff06dbd87048d09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:06.536229 32110 ts_tablet_manager.cc:1428] T 1ca64df3ab584cdea699ba758e88a0f7 P 875b3956c2304204aff06dbd87048d09: Time spent starting tablet: real 0.011s user 0.009s sys 0.004s
I20260810 00:45:06.536157 32111 raft_consensus.cc:348] T 1ca64df3ab584cdea699ba758e88a0f7 P 433319dcedfe4887857a97939ba14401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:06.536618 32111 raft_consensus.cc:374] T 1ca64df3ab584cdea699ba758e88a0f7 P 433319dcedfe4887857a97939ba14401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:06.536722 32111 raft_consensus.cc:729] T 1ca64df3ab584cdea699ba758e88a0f7 P 433319dcedfe4887857a97939ba14401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 433319dcedfe4887857a97939ba14401, State: Initialized, Role: FOLLOWER
I20260810 00:45:06.537287 32111 consensus_queue.cc:260] T 1ca64df3ab584cdea699ba758e88a0f7 P 433319dcedfe4887857a97939ba14401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:06.537388 32114 raft_consensus.cc:348] T 1ca64df3ab584cdea699ba758e88a0f7 P 658b70fbcba84147b8c6bb3dee887e7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:06.537777 32114 raft_consensus.cc:374] T 1ca64df3ab584cdea699ba758e88a0f7 P 658b70fbcba84147b8c6bb3dee887e7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:06.537976 32114 raft_consensus.cc:729] T 1ca64df3ab584cdea699ba758e88a0f7 P 658b70fbcba84147b8c6bb3dee887e7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 658b70fbcba84147b8c6bb3dee887e7d, State: Initialized, Role: FOLLOWER
I20260810 00:45:06.538604 32114 consensus_queue.cc:260] T 1ca64df3ab584cdea699ba758e88a0f7 P 658b70fbcba84147b8c6bb3dee887e7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:06.539098 32111 ts_tablet_manager.cc:1428] T 1ca64df3ab584cdea699ba758e88a0f7 P 433319dcedfe4887857a97939ba14401: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260810 00:45:06.539184 31880 heartbeater.cc:503] Master 127.28.6.62:41269 was elected leader, sending a full tablet report...
I20260810 00:45:06.539778 32078 heartbeater.cc:503] Master 127.28.6.62:41269 was elected leader, sending a full tablet report...
I20260810 00:45:06.540537 32114 ts_tablet_manager.cc:1428] T 1ca64df3ab584cdea699ba758e88a0f7 P 658b70fbcba84147b8c6bb3dee887e7d: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
W20260810 00:45:06.576443 31681 tablet.cc:2369] T 1ca64df3ab584cdea699ba758e88a0f7 P 875b3956c2304204aff06dbd87048d09: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 00:45:06.617026 31881 tablet.cc:2369] T 1ca64df3ab584cdea699ba758e88a0f7 P 433319dcedfe4887857a97939ba14401: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 00:45:06.644115 32080 tablet.cc:2369] T 1ca64df3ab584cdea699ba758e88a0f7 P 658b70fbcba84147b8c6bb3dee887e7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:45:06.664464 32118 raft_consensus.cc:482] T 1ca64df3ab584cdea699ba758e88a0f7 P 433319dcedfe4887857a97939ba14401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:45:06.664825 32118 raft_consensus.cc:504] T 1ca64df3ab584cdea699ba758e88a0f7 P 433319dcedfe4887857a97939ba14401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:06.666535 32118 leader_election.cc:288] T 1ca64df3ab584cdea699ba758e88a0f7 P 433319dcedfe4887857a97939ba14401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 875b3956c2304204aff06dbd87048d09 (127.28.6.1:45757), 658b70fbcba84147b8c6bb3dee887e7d (127.28.6.3:41205)
I20260810 00:45:06.674036 32105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:06.667099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:52999 (local address 127.28.6.1:45757)
0810 00:45:06.667337 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:06.667344 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:06.667678 (+ 334us) server_negotiation.cc:407] Connection header received
0810 00:45:06.667992 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:06.667998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:06.668095 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:06.668168 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:06.669104 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.670096 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.672072 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.673033 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.673134 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:06.673216 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:06.673653 (+ 437us) server_negotiation.cc:299] Negotiation successful
0810 00:45:06.673792 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 00:45:06.674045 32124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:06.666854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:45757 (local address 127.28.6.2:52999)
0810 00:45:06.667537 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:06.667586 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:06.667812 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:06.668409 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:06.668416 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:06.668441 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:06.668936 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:06.668953 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.670235 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.670241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:06.671902 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:06.671915 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.673266 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.673273 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:06.673445 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:06.673485 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 00:45:06.673599 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 00:45:06.673732 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":387,"threads_started":1}
I20260810 00:45:06.674516 32109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:06.668000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:35461 (local address 127.28.6.3:41205)
0810 00:45:06.668206 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:06.668213 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:06.668683 (+ 470us) server_negotiation.cc:407] Connection header received
0810 00:45:06.668965 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:06.668971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:06.669055 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:06.669138 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:06.669767 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.670663 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.672596 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.673544 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.673658 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:06.673770 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:06.674146 (+ 376us) server_negotiation.cc:299] Negotiation successful
0810 00:45:06.674285 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 00:45:06.674679 32125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:06.667751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:41205 (local address 127.28.6.2:35461)
0810 00:45:06.668569 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:06.668583 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:06.668780 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:06.669184 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:06.669189 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:06.669219 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:06.669583 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:06.669595 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.670819 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.670823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:06.672436 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:06.672451 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.673759 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.673765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:06.673945 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:06.673983 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 00:45:06.674096 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 00:45:06.674236 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":1,"thread_start_us":486,"threads_started":1}
I20260810 00:45:06.675309 31615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca64df3ab584cdea699ba758e88a0f7" candidate_uuid: "433319dcedfe4887857a97939ba14401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "875b3956c2304204aff06dbd87048d09" is_pre_election: true
I20260810 00:45:06.675696 32008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca64df3ab584cdea699ba758e88a0f7" candidate_uuid: "433319dcedfe4887857a97939ba14401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "658b70fbcba84147b8c6bb3dee887e7d" is_pre_election: true
I20260810 00:45:06.675805 31615 raft_consensus.cc:2454] T 1ca64df3ab584cdea699ba758e88a0f7 P 875b3956c2304204aff06dbd87048d09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 433319dcedfe4887857a97939ba14401 in term 0.
I20260810 00:45:06.676223 32008 raft_consensus.cc:2454] T 1ca64df3ab584cdea699ba758e88a0f7 P 658b70fbcba84147b8c6bb3dee887e7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 433319dcedfe4887857a97939ba14401 in term 0.
I20260810 00:45:06.676551 31713 leader_election.cc:302] T 1ca64df3ab584cdea699ba758e88a0f7 P 433319dcedfe4887857a97939ba14401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 433319dcedfe4887857a97939ba14401, 875b3956c2304204aff06dbd87048d09; no voters:
I20260810 00:45:06.676968 32118 raft_consensus.cc:2781] T 1ca64df3ab584cdea699ba758e88a0f7 P 433319dcedfe4887857a97939ba14401 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 00:45:06.677157 32118 raft_consensus.cc:482] T 1ca64df3ab584cdea699ba758e88a0f7 P 433319dcedfe4887857a97939ba14401 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 00:45:06.677284 32118 raft_consensus.cc:3037] T 1ca64df3ab584cdea699ba758e88a0f7 P 433319dcedfe4887857a97939ba14401 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:06.679654 32118 raft_consensus.cc:504] T 1ca64df3ab584cdea699ba758e88a0f7 P 433319dcedfe4887857a97939ba14401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:06.680282 32118 leader_election.cc:288] T 1ca64df3ab584cdea699ba758e88a0f7 P 433319dcedfe4887857a97939ba14401 [CANDIDATE]: Term 1 election: Requested vote from peers 875b3956c2304204aff06dbd87048d09 (127.28.6.1:45757), 658b70fbcba84147b8c6bb3dee887e7d (127.28.6.3:41205)
I20260810 00:45:06.680768 31615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca64df3ab584cdea699ba758e88a0f7" candidate_uuid: "433319dcedfe4887857a97939ba14401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "875b3956c2304204aff06dbd87048d09"
I20260810 00:45:06.680833 32008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca64df3ab584cdea699ba758e88a0f7" candidate_uuid: "433319dcedfe4887857a97939ba14401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "658b70fbcba84147b8c6bb3dee887e7d"
I20260810 00:45:06.681015 32008 raft_consensus.cc:3037] T 1ca64df3ab584cdea699ba758e88a0f7 P 658b70fbcba84147b8c6bb3dee887e7d [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:06.680999 31615 raft_consensus.cc:3037] T 1ca64df3ab584cdea699ba758e88a0f7 P 875b3956c2304204aff06dbd87048d09 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:06.684296 31615 raft_consensus.cc:2454] T 1ca64df3ab584cdea699ba758e88a0f7 P 875b3956c2304204aff06dbd87048d09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 433319dcedfe4887857a97939ba14401 in term 1.
I20260810 00:45:06.684290 32008 raft_consensus.cc:2454] T 1ca64df3ab584cdea699ba758e88a0f7 P 658b70fbcba84147b8c6bb3dee887e7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 433319dcedfe4887857a97939ba14401 in term 1.
I20260810 00:45:06.684686 31713 leader_election.cc:302] T 1ca64df3ab584cdea699ba758e88a0f7 P 433319dcedfe4887857a97939ba14401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 433319dcedfe4887857a97939ba14401, 875b3956c2304204aff06dbd87048d09; no voters:
I20260810 00:45:06.685003 32118 raft_consensus.cc:2781] T 1ca64df3ab584cdea699ba758e88a0f7 P 433319dcedfe4887857a97939ba14401 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:45:06.685446 32118 raft_consensus.cc:686] T 1ca64df3ab584cdea699ba758e88a0f7 P 433319dcedfe4887857a97939ba14401 [term 1 LEADER]: Becoming Leader. State: Replica: 433319dcedfe4887857a97939ba14401, State: Running, Role: LEADER
I20260810 00:45:06.685943 32118 consensus_queue.cc:237] T 1ca64df3ab584cdea699ba758e88a0f7 P 433319dcedfe4887857a97939ba14401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:06.690613 31369 catalog_manager.cc:5495] T 1ca64df3ab584cdea699ba758e88a0f7 P 433319dcedfe4887857a97939ba14401 reported cstate change: term changed from 0 to 1, leader changed from <none> to 433319dcedfe4887857a97939ba14401 (127.28.6.2). New cstate: current_term: 1 leader_uuid: "433319dcedfe4887857a97939ba14401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } health_report { overall_health: UNKNOWN } } }
Aug 10 00:45:06 dist-test-slave-1nl6 krb5kdc[31306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322704, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:45:06.759114 31511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:06.745464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42166 (local address 127.28.6.62:41269)
0810 00:45:06.745688 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:06.745693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:06.746198 (+ 505us) server_negotiation.cc:407] Connection header received
0810 00:45:06.746424 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:06.746428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:06.746482 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:06.746581 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:06.747859 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.748844 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.750131 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.750550 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.750707 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:06.753975 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:06.753980 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:06.753983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:06.754011 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:06.756434 (+ 2423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:06.757113 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:06.757121 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:06.757124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:06.757262 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:06.757624 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:06.757631 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:06.757636 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:06.757890 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:06.758043 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:06.758458 (+ 415us) server_negotiation.cc:299] Negotiation successful
0810 00:45:06.758775 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":118,"server-negotiator.queue_time_us":66}
I20260810 00:45:06.759136 32135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:06.745476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:41269 (local address 127.0.0.1:42166)
0810 00:45:06.746092 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:06.746108 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:06.746285 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:06.746606 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:06.746610 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:06.747434 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:06.747706 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:06.747718 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.748980 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.748988 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:06.749988 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:06.750000 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.750740 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.750745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:06.750893 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:06.751586 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:06.751604 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:06.753798 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:06.756574 (+ 2776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:06.756585 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:06.756593 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:06.756997 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:06.757371 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:06.757377 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:06.757381 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:06.757517 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:06.758020 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:06.758028 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:06.758188 (+ 160us) client_negotiation.cc:769] Sending connection context
0810 00:45:06.758365 (+ 177us) client_negotiation.cc:240] Negotiation successful
0810 00:45:06.758640 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":373,"threads_started":1}
I20260810 00:45:06.764422 31369 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:42166:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 00:45:06.765415 31369 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 00:45:06.772219 31787 tablet_service.cc:1467] Processing CreateTablet for tablet e454e20291954c60b231eaa681c3e91d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ad95613c70e74e6780112d3d19de0744]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 00:45:06.772178 31575 tablet_service.cc:1467] Processing CreateTablet for tablet e454e20291954c60b231eaa681c3e91d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ad95613c70e74e6780112d3d19de0744]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 00:45:06.772564 31975 tablet_service.cc:1467] Processing CreateTablet for tablet e454e20291954c60b231eaa681c3e91d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ad95613c70e74e6780112d3d19de0744]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 00:45:06.773087 31787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e454e20291954c60b231eaa681c3e91d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:06.773115 31575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e454e20291954c60b231eaa681c3e91d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:06.773362 31975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e454e20291954c60b231eaa681c3e91d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:06.779019 32111 tablet_bootstrap.cc:492] T e454e20291954c60b231eaa681c3e91d P 433319dcedfe4887857a97939ba14401: Bootstrap starting.
I20260810 00:45:06.780377 32114 tablet_bootstrap.cc:492] T e454e20291954c60b231eaa681c3e91d P 658b70fbcba84147b8c6bb3dee887e7d: Bootstrap starting.
I20260810 00:45:06.780387 32110 tablet_bootstrap.cc:492] T e454e20291954c60b231eaa681c3e91d P 875b3956c2304204aff06dbd87048d09: Bootstrap starting.
I20260810 00:45:06.780548 32111 tablet_bootstrap.cc:654] T e454e20291954c60b231eaa681c3e91d P 433319dcedfe4887857a97939ba14401: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:06.782096 32114 tablet_bootstrap.cc:654] T e454e20291954c60b231eaa681c3e91d P 658b70fbcba84147b8c6bb3dee887e7d: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:06.782193 32110 tablet_bootstrap.cc:654] T e454e20291954c60b231eaa681c3e91d P 875b3956c2304204aff06dbd87048d09: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:06.783442 32111 tablet_bootstrap.cc:492] T e454e20291954c60b231eaa681c3e91d P 433319dcedfe4887857a97939ba14401: No bootstrap required, opened a new log
I20260810 00:45:06.783600 32111 ts_tablet_manager.cc:1397] T e454e20291954c60b231eaa681c3e91d P 433319dcedfe4887857a97939ba14401: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 00:45:06.784441 32111 raft_consensus.cc:348] T e454e20291954c60b231eaa681c3e91d P 433319dcedfe4887857a97939ba14401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:06.784615 32114 tablet_bootstrap.cc:492] T e454e20291954c60b231eaa681c3e91d P 658b70fbcba84147b8c6bb3dee887e7d: No bootstrap required, opened a new log
I20260810 00:45:06.784652 32111 raft_consensus.cc:374] T e454e20291954c60b231eaa681c3e91d P 433319dcedfe4887857a97939ba14401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:06.784787 32114 ts_tablet_manager.cc:1397] T e454e20291954c60b231eaa681c3e91d P 658b70fbcba84147b8c6bb3dee887e7d: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260810 00:45:06.784795 32111 raft_consensus.cc:729] T e454e20291954c60b231eaa681c3e91d P 433319dcedfe4887857a97939ba14401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 433319dcedfe4887857a97939ba14401, State: Initialized, Role: FOLLOWER
I20260810 00:45:06.785044 32111 consensus_queue.cc:260] T e454e20291954c60b231eaa681c3e91d P 433319dcedfe4887857a97939ba14401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:06.785356 32110 tablet_bootstrap.cc:492] T e454e20291954c60b231eaa681c3e91d P 875b3956c2304204aff06dbd87048d09: No bootstrap required, opened a new log
I20260810 00:45:06.785531 32110 ts_tablet_manager.cc:1397] T e454e20291954c60b231eaa681c3e91d P 875b3956c2304204aff06dbd87048d09: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260810 00:45:06.785495 32118 tablet_replica.cc:439] T e454e20291954c60b231eaa681c3e91d P 433319dcedfe4887857a97939ba14401: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } } }
I20260810 00:45:06.785566 32114 raft_consensus.cc:348] T e454e20291954c60b231eaa681c3e91d P 658b70fbcba84147b8c6bb3dee887e7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:06.785786 32118 tablet_replica.cc:442] T e454e20291954c60b231eaa681c3e91d P 433319dcedfe4887857a97939ba14401: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 00:45:06.785835 32114 raft_consensus.cc:374] T e454e20291954c60b231eaa681c3e91d P 658b70fbcba84147b8c6bb3dee887e7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:06.785925 32114 raft_consensus.cc:729] T e454e20291954c60b231eaa681c3e91d P 658b70fbcba84147b8c6bb3dee887e7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 658b70fbcba84147b8c6bb3dee887e7d, State: Initialized, Role: FOLLOWER
I20260810 00:45:06.786175 32114 consensus_queue.cc:260] T e454e20291954c60b231eaa681c3e91d P 658b70fbcba84147b8c6bb3dee887e7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:06.786453 32111 ts_tablet_manager.cc:1428] T e454e20291954c60b231eaa681c3e91d P 433319dcedfe4887857a97939ba14401: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260810 00:45:06.786341 32110 raft_consensus.cc:348] T e454e20291954c60b231eaa681c3e91d P 875b3956c2304204aff06dbd87048d09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:06.786572 32110 raft_consensus.cc:374] T e454e20291954c60b231eaa681c3e91d P 875b3956c2304204aff06dbd87048d09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:06.786720 32110 raft_consensus.cc:729] T e454e20291954c60b231eaa681c3e91d P 875b3956c2304204aff06dbd87048d09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 875b3956c2304204aff06dbd87048d09, State: Initialized, Role: FOLLOWER
I20260810 00:45:06.786684 32123 tablet_replica.cc:439] T e454e20291954c60b231eaa681c3e91d P 658b70fbcba84147b8c6bb3dee887e7d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } } }
I20260810 00:45:06.787081 32123 tablet_replica.cc:442] T e454e20291954c60b231eaa681c3e91d P 658b70fbcba84147b8c6bb3dee887e7d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 00:45:06.786978 32110 consensus_queue.cc:260] T e454e20291954c60b231eaa681c3e91d P 875b3956c2304204aff06dbd87048d09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:06.787424 32117 tablet_replica.cc:439] T e454e20291954c60b231eaa681c3e91d P 875b3956c2304204aff06dbd87048d09: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } } }
I20260810 00:45:06.787778 32117 tablet_replica.cc:442] T e454e20291954c60b231eaa681c3e91d P 875b3956c2304204aff06dbd87048d09: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 00:45:06.787616 32110 ts_tablet_manager.cc:1428] T e454e20291954c60b231eaa681c3e91d P 875b3956c2304204aff06dbd87048d09: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260810 00:45:06.788826 32114 ts_tablet_manager.cc:1428] T e454e20291954c60b231eaa681c3e91d P 658b70fbcba84147b8c6bb3dee887e7d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260810 00:45:06.923080 32123 raft_consensus.cc:482] T e454e20291954c60b231eaa681c3e91d P 658b70fbcba84147b8c6bb3dee887e7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:45:06.923334 32123 raft_consensus.cc:504] T e454e20291954c60b231eaa681c3e91d P 658b70fbcba84147b8c6bb3dee887e7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:06.924834 32123 leader_election.cc:288] T e454e20291954c60b231eaa681c3e91d P 658b70fbcba84147b8c6bb3dee887e7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 875b3956c2304204aff06dbd87048d09 (127.28.6.1:45757), 433319dcedfe4887857a97939ba14401 (127.28.6.2:46687)
I20260810 00:45:06.933221 32101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:06.925246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:56085 (local address 127.28.6.2:46687)
0810 00:45:06.925447 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:06.925452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:06.926012 (+ 560us) server_negotiation.cc:407] Connection header received
0810 00:45:06.926270 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:06.926275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:06.926373 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:06.926441 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:06.927282 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.928340 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.930871 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.932188 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.932343 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:06.932458 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:06.932808 (+ 350us) server_negotiation.cc:299] Negotiation successful
0810 00:45:06.932949 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 00:45:06.933269 32105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:06.925255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:50095 (local address 127.28.6.1:45757)
0810 00:45:06.925446 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:06.925453 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:06.926028 (+ 575us) server_negotiation.cc:407] Connection header received
0810 00:45:06.926231 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:06.926236 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:06.926335 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:06.926444 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:06.926927 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.928131 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.930872 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.932203 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.932343 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:06.932458 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:06.932808 (+ 350us) server_negotiation.cc:299] Negotiation successful
0810 00:45:06.932949 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 00:45:06.933398 32139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:06.925115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:45757 (local address 127.28.6.3:50095)
0810 00:45:06.925883 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:06.925904 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:06.926063 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:06.926468 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:06.926473 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:06.926507 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:06.926711 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:06.926720 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.928272 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.928276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:06.930661 (+ 2385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:06.930671 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.932420 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.932426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:06.932632 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:06.932656 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 00:45:06.932756 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 00:45:06.932955 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"mutex_wait_us":175,"thread_start_us":433,"threads_started":1}
I20260810 00:45:06.933676 32140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:06.925115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:46687 (local address 127.28.6.3:56085)
0810 00:45:06.925915 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:06.925932 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:06.926087 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:06.926677 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:06.926681 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:06.926706 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:06.927083 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:06.927099 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.928457 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.928462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:06.930656 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:06.930671 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.932381 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.932389 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:06.932618 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:06.932655 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 00:45:06.932756 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 00:45:06.932955 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"mutex_wait_us":160,"thread_start_us":391,"threads_started":1}
I20260810 00:45:06.934044 31615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e454e20291954c60b231eaa681c3e91d" candidate_uuid: "658b70fbcba84147b8c6bb3dee887e7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "875b3956c2304204aff06dbd87048d09" is_pre_election: true
I20260810 00:45:06.934391 31615 raft_consensus.cc:2454] T e454e20291954c60b231eaa681c3e91d P 875b3956c2304204aff06dbd87048d09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 658b70fbcba84147b8c6bb3dee887e7d in term 0.
I20260810 00:45:06.934628 31822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e454e20291954c60b231eaa681c3e91d" candidate_uuid: "658b70fbcba84147b8c6bb3dee887e7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "433319dcedfe4887857a97939ba14401" is_pre_election: true
I20260810 00:45:06.934878 31908 leader_election.cc:302] T e454e20291954c60b231eaa681c3e91d P 658b70fbcba84147b8c6bb3dee887e7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 658b70fbcba84147b8c6bb3dee887e7d, 875b3956c2304204aff06dbd87048d09; no voters:
I20260810 00:45:06.935176 31822 raft_consensus.cc:2454] T e454e20291954c60b231eaa681c3e91d P 433319dcedfe4887857a97939ba14401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 658b70fbcba84147b8c6bb3dee887e7d in term 0.
I20260810 00:45:06.935369 32123 raft_consensus.cc:2781] T e454e20291954c60b231eaa681c3e91d P 658b70fbcba84147b8c6bb3dee887e7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 00:45:06.935535 32123 raft_consensus.cc:482] T e454e20291954c60b231eaa681c3e91d P 658b70fbcba84147b8c6bb3dee887e7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 00:45:06.935684 32123 raft_consensus.cc:3037] T e454e20291954c60b231eaa681c3e91d P 658b70fbcba84147b8c6bb3dee887e7d [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:06.937909 32123 raft_consensus.cc:504] T e454e20291954c60b231eaa681c3e91d P 658b70fbcba84147b8c6bb3dee887e7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:06.938619 32123 leader_election.cc:288] T e454e20291954c60b231eaa681c3e91d P 658b70fbcba84147b8c6bb3dee887e7d [CANDIDATE]: Term 1 election: Requested vote from peers 875b3956c2304204aff06dbd87048d09 (127.28.6.1:45757), 433319dcedfe4887857a97939ba14401 (127.28.6.2:46687)
I20260810 00:45:06.938977 31615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e454e20291954c60b231eaa681c3e91d" candidate_uuid: "658b70fbcba84147b8c6bb3dee887e7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "875b3956c2304204aff06dbd87048d09"
I20260810 00:45:06.939234 31615 raft_consensus.cc:3037] T e454e20291954c60b231eaa681c3e91d P 875b3956c2304204aff06dbd87048d09 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:06.939184 31822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e454e20291954c60b231eaa681c3e91d" candidate_uuid: "658b70fbcba84147b8c6bb3dee887e7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "433319dcedfe4887857a97939ba14401"
I20260810 00:45:06.939363 31822 raft_consensus.cc:3037] T e454e20291954c60b231eaa681c3e91d P 433319dcedfe4887857a97939ba14401 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:06.942299 31615 raft_consensus.cc:2454] T e454e20291954c60b231eaa681c3e91d P 875b3956c2304204aff06dbd87048d09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 658b70fbcba84147b8c6bb3dee887e7d in term 1.
I20260810 00:45:06.942404 31822 raft_consensus.cc:2454] T e454e20291954c60b231eaa681c3e91d P 433319dcedfe4887857a97939ba14401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 658b70fbcba84147b8c6bb3dee887e7d in term 1.
I20260810 00:45:06.942785 31908 leader_election.cc:302] T e454e20291954c60b231eaa681c3e91d P 658b70fbcba84147b8c6bb3dee887e7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 658b70fbcba84147b8c6bb3dee887e7d, 875b3956c2304204aff06dbd87048d09; no voters:
I20260810 00:45:06.943075 32123 raft_consensus.cc:2781] T e454e20291954c60b231eaa681c3e91d P 658b70fbcba84147b8c6bb3dee887e7d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:45:06.943490 32123 raft_consensus.cc:686] T e454e20291954c60b231eaa681c3e91d P 658b70fbcba84147b8c6bb3dee887e7d [term 1 LEADER]: Becoming Leader. State: Replica: 658b70fbcba84147b8c6bb3dee887e7d, State: Running, Role: LEADER
I20260810 00:45:06.943958 32123 consensus_queue.cc:237] T e454e20291954c60b231eaa681c3e91d P 658b70fbcba84147b8c6bb3dee887e7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:06.947538 32141 tablet_replica.cc:439] T e454e20291954c60b231eaa681c3e91d P 658b70fbcba84147b8c6bb3dee887e7d: TxnStatusTablet state changed. Reason: New leader 658b70fbcba84147b8c6bb3dee887e7d. Latest consensus state: current_term: 1 leader_uuid: "658b70fbcba84147b8c6bb3dee887e7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } } }
I20260810 00:45:06.947925 32141 tablet_replica.cc:442] T e454e20291954c60b231eaa681c3e91d P 658b70fbcba84147b8c6bb3dee887e7d: This TxnStatusTablet replica's current role is: LEADER
I20260810 00:45:06.948724 32143 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 00:45:06.949024 32143 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 00:45:06.949342 31372 catalog_manager.cc:5495] T e454e20291954c60b231eaa681c3e91d P 658b70fbcba84147b8c6bb3dee887e7d reported cstate change: term changed from 0 to 1, leader changed from <none> to 658b70fbcba84147b8c6bb3dee887e7d (127.28.6.3). New cstate: current_term: 1 leader_uuid: "658b70fbcba84147b8c6bb3dee887e7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } health_report { overall_health: HEALTHY } } }
I20260810 00:45:06.978546 32135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:06.972283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:41205 (local address 127.0.0.1:36794)
0810 00:45:06.972516 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:06.972538 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:06.972672 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:06.973174 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:06.973177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:06.973187 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 00:45:06.973366 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:06.973373 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.974860 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.974863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:06.975896 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:06.975909 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.976873 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.976885 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:06.977031 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:06.977084 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 00:45:06.978110 (+ 1026us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 00:45:06.978128 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 00:45:06.978200 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 00:45:06.978313 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 00:45:06.978641 32109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:06.972288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36794 (local address 127.28.6.3:41205)
0810 00:45:06.972469 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:06.972477 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:06.972622 (+ 145us) server_negotiation.cc:407] Connection header received
0810 00:45:06.972860 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:06.972868 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:06.972992 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:06.973169 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:45:06.973502 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.974726 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.976040 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:06.976689 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:06.976850 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:06.977283 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:45:06.978000 (+ 717us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:45:06.978150 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:06.978226 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 00:45:06.978313 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 00:45:06.987195 31615 raft_consensus.cc:1261] T e454e20291954c60b231eaa681c3e91d P 875b3956c2304204aff06dbd87048d09 [term 1 FOLLOWER]: Refusing update from remote peer 658b70fbcba84147b8c6bb3dee887e7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:45:06.987203 31822 raft_consensus.cc:1261] T e454e20291954c60b231eaa681c3e91d P 433319dcedfe4887857a97939ba14401 [term 1 FOLLOWER]: Refusing update from remote peer 658b70fbcba84147b8c6bb3dee887e7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:45:06.988234 32123 consensus_queue.cc:1035] T e454e20291954c60b231eaa681c3e91d P 658b70fbcba84147b8c6bb3dee887e7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:45:06.988541 32141 consensus_queue.cc:1035] T e454e20291954c60b231eaa681c3e91d P 658b70fbcba84147b8c6bb3dee887e7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:45:06.998973 32118 tablet_replica.cc:439] T e454e20291954c60b231eaa681c3e91d P 433319dcedfe4887857a97939ba14401: TxnStatusTablet state changed. Reason: New leader 658b70fbcba84147b8c6bb3dee887e7d. Latest consensus state: current_term: 1 leader_uuid: "658b70fbcba84147b8c6bb3dee887e7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } } }
I20260810 00:45:06.999342 32118 tablet_replica.cc:442] T e454e20291954c60b231eaa681c3e91d P 433319dcedfe4887857a97939ba14401: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 00:45:07.004336 32147 mvcc.cc:205] Tried to move back new op lower bound from 7316777807809515520 to 7316777807645986816. Current Snapshot: MvccSnapshot[applied={T|T < 7316777807809515520}]
I20260810 00:45:07.005009 32141 tablet_replica.cc:439] T e454e20291954c60b231eaa681c3e91d P 658b70fbcba84147b8c6bb3dee887e7d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "658b70fbcba84147b8c6bb3dee887e7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } } }
I20260810 00:45:07.005280 32141 tablet_replica.cc:442] T e454e20291954c60b231eaa681c3e91d P 658b70fbcba84147b8c6bb3dee887e7d: This TxnStatusTablet replica's current role is: LEADER
I20260810 00:45:07.006934 32118 tablet_replica.cc:439] T e454e20291954c60b231eaa681c3e91d P 433319dcedfe4887857a97939ba14401: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "658b70fbcba84147b8c6bb3dee887e7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } } }
I20260810 00:45:07.007198 32118 tablet_replica.cc:442] T e454e20291954c60b231eaa681c3e91d P 433319dcedfe4887857a97939ba14401: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 00:45:07.014161 32117 tablet_replica.cc:439] T e454e20291954c60b231eaa681c3e91d P 875b3956c2304204aff06dbd87048d09: TxnStatusTablet state changed. Reason: New leader 658b70fbcba84147b8c6bb3dee887e7d. Latest consensus state: current_term: 1 leader_uuid: "658b70fbcba84147b8c6bb3dee887e7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } } }
I20260810 00:45:07.014488 32117 tablet_replica.cc:442] T e454e20291954c60b231eaa681c3e91d P 875b3956c2304204aff06dbd87048d09: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 00:45:07.014569 32123 tablet_replica.cc:439] T e454e20291954c60b231eaa681c3e91d P 658b70fbcba84147b8c6bb3dee887e7d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "658b70fbcba84147b8c6bb3dee887e7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } } }
I20260810 00:45:07.014860 32123 tablet_replica.cc:442] T e454e20291954c60b231eaa681c3e91d P 658b70fbcba84147b8c6bb3dee887e7d: This TxnStatusTablet replica's current role is: LEADER
I20260810 00:45:07.017957 32149 mvcc.cc:205] Tried to move back new op lower bound from 7316777807809515520 to 7316777807645986816. Current Snapshot: MvccSnapshot[applied={T|T < 7316777807809515520}]
I20260810 00:45:07.018613 32117 tablet_replica.cc:439] T e454e20291954c60b231eaa681c3e91d P 875b3956c2304204aff06dbd87048d09: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "658b70fbcba84147b8c6bb3dee887e7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } } }
I20260810 00:45:07.018848 32117 tablet_replica.cc:442] T e454e20291954c60b231eaa681c3e91d P 875b3956c2304204aff06dbd87048d09: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 00:45:07.024277 32101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:07.012184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48336 (local address 127.28.6.2:46687)
0810 00:45:07.012361 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:07.012370 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:07.012862 (+ 492us) server_negotiation.cc:407] Connection header received
0810 00:45:07.013132 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:07.013137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:07.013250 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:07.013401 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:45:07.015449 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:07.016540 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:07.021525 (+ 4985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:07.022173 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:07.022315 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:07.022708 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:45:07.023415 (+ 707us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:45:07.023770 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:07.023830 (+ 60us) server_negotiation.cc:299] Negotiation successful
0810 00:45:07.023920 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 00:45:07.041630 32162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:07.035305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:41205 (local address 127.28.6.2:48109)
0810 00:45:07.035999 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:07.036015 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:07.036172 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:07.036548 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:07.036551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:07.036560 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 00:45:07.036855 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:07.036863 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:07.038240 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:07.038243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:07.039388 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:07.039401 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:07.040432 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:07.040436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:07.040568 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:07.040625 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 00:45:07.041055 (+ 430us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 00:45:07.041063 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 00:45:07.041119 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 00:45:07.041260 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":351,"threads_started":1}
I20260810 00:45:07.041747 32109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:07.035334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:48109 (local address 127.28.6.3:41205)
0810 00:45:07.035537 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:07.035549 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:07.036093 (+ 544us) server_negotiation.cc:407] Connection header received
0810 00:45:07.036325 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:07.036331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:07.036435 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:07.036533 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:45:07.037006 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:07.038114 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:07.039529 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:07.040220 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:07.040348 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:07.040818 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:45:07.040951 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:45:07.041034 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:07.041198 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 00:45:07.041287 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 00:45:07.051605 31615 raft_consensus.cc:1261] T 1ca64df3ab584cdea699ba758e88a0f7 P 875b3956c2304204aff06dbd87048d09 [term 1 FOLLOWER]: Refusing update from remote peer 433319dcedfe4887857a97939ba14401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:45:07.051769 32008 raft_consensus.cc:1261] T 1ca64df3ab584cdea699ba758e88a0f7 P 658b70fbcba84147b8c6bb3dee887e7d [term 1 FOLLOWER]: Refusing update from remote peer 433319dcedfe4887857a97939ba14401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:45:07.052350 32126 consensus_queue.cc:1035] T 1ca64df3ab584cdea699ba758e88a0f7 P 433319dcedfe4887857a97939ba14401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:45:07.052938 32118 consensus_queue.cc:1035] T 1ca64df3ab584cdea699ba758e88a0f7 P 433319dcedfe4887857a97939ba14401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:45:07.094815 32101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:07.087016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:56399 (local address 127.28.6.2:46687)
0810 00:45:07.087428 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:07.087435 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:07.087985 (+ 550us) server_negotiation.cc:407] Connection header received
0810 00:45:07.088596 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:07.088602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:07.088709 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:07.088778 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:45:07.089600 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:07.090309 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:07.093068 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:07.093517 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:07.093651 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:07.094093 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:45:07.094197 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:45:07.094297 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:07.094452 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 00:45:07.094530 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277}
I20260810 00:45:07.096046 32171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:07.087015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:46687 (local address 127.28.6.3:56399)
0810 00:45:07.087881 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:07.087906 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:07.088422 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:07.089074 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:07.089078 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:07.089088 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 00:45:07.089434 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:07.089443 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:07.090455 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:07.090460 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:07.092926 (+ 2466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:07.092939 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:07.093722 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:07.093729 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:07.093857 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:07.093912 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 00:45:07.094302 (+ 390us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 00:45:07.094317 (+ 15us) client_negotiation.cc:769] Sending connection context
0810 00:45:07.094414 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 00:45:07.094558 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":429,"threads_started":1}
I20260810 00:45:07.133455 31372 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:45:07.133757 31372 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:45:07.139756 31372 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7678fc95ce74a4ba1a31ad544a71001: Sending DeleteTablet for 3 replicas of tablet 1ca64df3ab584cdea699ba758e88a0f7
I20260810 00:45:07.144037 31575 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ca64df3ab584cdea699ba758e88a0f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:45:07 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:39678
I20260810 00:45:07.146682 31975 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ca64df3ab584cdea699ba758e88a0f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:45:07 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:36782
I20260810 00:45:07.150291 32175 tablet_replica.cc:330] T 1ca64df3ab584cdea699ba758e88a0f7 P 875b3956c2304204aff06dbd87048d09: stopping tablet replica
I20260810 00:45:07.150717 32175 raft_consensus.cc:2229] T 1ca64df3ab584cdea699ba758e88a0f7 P 875b3956c2304204aff06dbd87048d09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:45:07.151118 32175 raft_consensus.cc:2258] T 1ca64df3ab584cdea699ba758e88a0f7 P 875b3956c2304204aff06dbd87048d09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:45:07.154284 31787 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ca64df3ab584cdea699ba758e88a0f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:45:07 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:48322
I20260810 00:45:07.154415 32175 ts_tablet_manager.cc:1905] T 1ca64df3ab584cdea699ba758e88a0f7 P 875b3956c2304204aff06dbd87048d09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:45:07.158383 31372 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42160:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 00:45:07.160553 31372 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 00:45:07.161950 32175 ts_tablet_manager.cc:1918] T 1ca64df3ab584cdea699ba758e88a0f7 P 875b3956c2304204aff06dbd87048d09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 00:45:07.162119 32175 log.cc:1196] T 1ca64df3ab584cdea699ba758e88a0f7 P 875b3956c2304204aff06dbd87048d09: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-0/wal/wals/1ca64df3ab584cdea699ba758e88a0f7
I20260810 00:45:07.162649 32175 ts_tablet_manager.cc:1939] T 1ca64df3ab584cdea699ba758e88a0f7 P 875b3956c2304204aff06dbd87048d09: Deleting consensus metadata
I20260810 00:45:07.165163 31352 catalog_manager.cc:4841] TS 875b3956c2304204aff06dbd87048d09 (127.28.6.1:45757): tablet 1ca64df3ab584cdea699ba758e88a0f7 (table test-table [id=2213e988d3f94fd88ec9c34dd970eda1]) successfully deleted
I20260810 00:45:07.166033 32176 tablet_replica.cc:330] T 1ca64df3ab584cdea699ba758e88a0f7 P 658b70fbcba84147b8c6bb3dee887e7d: stopping tablet replica
I20260810 00:45:07.166509 32176 raft_consensus.cc:2229] T 1ca64df3ab584cdea699ba758e88a0f7 P 658b70fbcba84147b8c6bb3dee887e7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:45:07.166888 32176 raft_consensus.cc:2258] T 1ca64df3ab584cdea699ba758e88a0f7 P 658b70fbcba84147b8c6bb3dee887e7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:45:07.168849 32176 ts_tablet_manager.cc:1905] T 1ca64df3ab584cdea699ba758e88a0f7 P 658b70fbcba84147b8c6bb3dee887e7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:45:07.174175 31575 tablet_service.cc:1467] Processing CreateTablet for tablet e8c843aa5cb24c8d9eb824bc2dee05b0 (DEFAULT_TABLE table=test-table [id=1391b8fce3f24897afe09e32574496d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:07.174620 31575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8c843aa5cb24c8d9eb824bc2dee05b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:07.177531 32176 ts_tablet_manager.cc:1918] T 1ca64df3ab584cdea699ba758e88a0f7 P 658b70fbcba84147b8c6bb3dee887e7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 00:45:07.177712 32176 log.cc:1196] T 1ca64df3ab584cdea699ba758e88a0f7 P 658b70fbcba84147b8c6bb3dee887e7d: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-2/wal/wals/1ca64df3ab584cdea699ba758e88a0f7
I20260810 00:45:07.178265 32176 ts_tablet_manager.cc:1939] T 1ca64df3ab584cdea699ba758e88a0f7 P 658b70fbcba84147b8c6bb3dee887e7d: Deleting consensus metadata
I20260810 00:45:07.179737 31786 tablet_service.cc:1467] Processing CreateTablet for tablet e8c843aa5cb24c8d9eb824bc2dee05b0 (DEFAULT_TABLE table=test-table [id=1391b8fce3f24897afe09e32574496d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:07.180323 31786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8c843aa5cb24c8d9eb824bc2dee05b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:07.185930 32110 tablet_bootstrap.cc:492] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 875b3956c2304204aff06dbd87048d09: Bootstrap starting.
I20260810 00:45:07.187055 32111 tablet_bootstrap.cc:492] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 433319dcedfe4887857a97939ba14401: Bootstrap starting.
I20260810 00:45:07.187702 32110 tablet_bootstrap.cc:654] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 875b3956c2304204aff06dbd87048d09: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:07.188931 32111 tablet_bootstrap.cc:654] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 433319dcedfe4887857a97939ba14401: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:07.190762 31352 catalog_manager.cc:4841] TS 658b70fbcba84147b8c6bb3dee887e7d (127.28.6.3:41205): tablet 1ca64df3ab584cdea699ba758e88a0f7 (table test-table [id=2213e988d3f94fd88ec9c34dd970eda1]) successfully deleted
I20260810 00:45:07.192061 31975 tablet_service.cc:1467] Processing CreateTablet for tablet e8c843aa5cb24c8d9eb824bc2dee05b0 (DEFAULT_TABLE table=test-table [id=1391b8fce3f24897afe09e32574496d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:07.192550 31975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8c843aa5cb24c8d9eb824bc2dee05b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:07.193025 32110 tablet_bootstrap.cc:492] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 875b3956c2304204aff06dbd87048d09: No bootstrap required, opened a new log
I20260810 00:45:07.193198 32110 ts_tablet_manager.cc:1397] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 875b3956c2304204aff06dbd87048d09: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260810 00:45:07.193976 32110 raft_consensus.cc:348] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 875b3956c2304204aff06dbd87048d09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:07.194264 32110 raft_consensus.cc:374] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 875b3956c2304204aff06dbd87048d09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:07.194355 32110 raft_consensus.cc:729] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 875b3956c2304204aff06dbd87048d09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 875b3956c2304204aff06dbd87048d09, State: Initialized, Role: FOLLOWER
I20260810 00:45:07.194701 32110 consensus_queue.cc:260] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 875b3956c2304204aff06dbd87048d09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:07.195230 32111 tablet_bootstrap.cc:492] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 433319dcedfe4887857a97939ba14401: No bootstrap required, opened a new log
I20260810 00:45:07.195386 32111 ts_tablet_manager.cc:1397] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 433319dcedfe4887857a97939ba14401: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260810 00:45:07.196260 32111 raft_consensus.cc:348] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 433319dcedfe4887857a97939ba14401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:07.196520 32111 raft_consensus.cc:374] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 433319dcedfe4887857a97939ba14401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:07.196594 32111 raft_consensus.cc:729] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 433319dcedfe4887857a97939ba14401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 433319dcedfe4887857a97939ba14401, State: Initialized, Role: FOLLOWER
I20260810 00:45:07.196835 32111 consensus_queue.cc:260] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 433319dcedfe4887857a97939ba14401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:07.197773 32110 ts_tablet_manager.cc:1428] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 875b3956c2304204aff06dbd87048d09: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260810 00:45:07.199828 32111 ts_tablet_manager.cc:1428] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 433319dcedfe4887857a97939ba14401: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260810 00:45:07.205663 32114 tablet_bootstrap.cc:492] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 658b70fbcba84147b8c6bb3dee887e7d: Bootstrap starting.
I20260810 00:45:07.207404 32114 tablet_bootstrap.cc:654] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 658b70fbcba84147b8c6bb3dee887e7d: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:07.209322 32177 tablet_replica.cc:330] T 1ca64df3ab584cdea699ba758e88a0f7 P 433319dcedfe4887857a97939ba14401: stopping tablet replica
I20260810 00:45:07.209867 32177 raft_consensus.cc:2229] T 1ca64df3ab584cdea699ba758e88a0f7 P 433319dcedfe4887857a97939ba14401 [term 1 LEADER]: Raft consensus shutting down.
I20260810 00:45:07.210549 32177 raft_consensus.cc:2258] T 1ca64df3ab584cdea699ba758e88a0f7 P 433319dcedfe4887857a97939ba14401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:45:07.212384 32114 tablet_bootstrap.cc:492] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 658b70fbcba84147b8c6bb3dee887e7d: No bootstrap required, opened a new log
I20260810 00:45:07.212556 32114 ts_tablet_manager.cc:1397] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 658b70fbcba84147b8c6bb3dee887e7d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 00:45:07.213296 32114 raft_consensus.cc:348] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 658b70fbcba84147b8c6bb3dee887e7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:07.213553 32114 raft_consensus.cc:374] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 658b70fbcba84147b8c6bb3dee887e7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:07.213632 32114 raft_consensus.cc:729] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 658b70fbcba84147b8c6bb3dee887e7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 658b70fbcba84147b8c6bb3dee887e7d, State: Initialized, Role: FOLLOWER
I20260810 00:45:07.213909 32114 consensus_queue.cc:260] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 658b70fbcba84147b8c6bb3dee887e7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:07.214589 32114 ts_tablet_manager.cc:1428] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 658b70fbcba84147b8c6bb3dee887e7d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 00:45:07.219905 32177 ts_tablet_manager.cc:1905] T 1ca64df3ab584cdea699ba758e88a0f7 P 433319dcedfe4887857a97939ba14401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:45:07.224905 32166 raft_consensus.cc:482] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 433319dcedfe4887857a97939ba14401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:45:07.225078 32166 raft_consensus.cc:504] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 433319dcedfe4887857a97939ba14401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:07.226065 32166 leader_election.cc:288] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 433319dcedfe4887857a97939ba14401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 875b3956c2304204aff06dbd87048d09 (127.28.6.1:45757), 658b70fbcba84147b8c6bb3dee887e7d (127.28.6.3:41205)
I20260810 00:45:07.226827 31612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c843aa5cb24c8d9eb824bc2dee05b0" candidate_uuid: "433319dcedfe4887857a97939ba14401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "875b3956c2304204aff06dbd87048d09" is_pre_election: true
I20260810 00:45:07.227134 31612 raft_consensus.cc:2454] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 875b3956c2304204aff06dbd87048d09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 433319dcedfe4887857a97939ba14401 in term 0.
I20260810 00:45:07.228029 32008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c843aa5cb24c8d9eb824bc2dee05b0" candidate_uuid: "433319dcedfe4887857a97939ba14401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "658b70fbcba84147b8c6bb3dee887e7d" is_pre_election: true
I20260810 00:45:07.228260 32008 raft_consensus.cc:2454] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 658b70fbcba84147b8c6bb3dee887e7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 433319dcedfe4887857a97939ba14401 in term 0.
I20260810 00:45:07.228717 32177 ts_tablet_manager.cc:1918] T 1ca64df3ab584cdea699ba758e88a0f7 P 433319dcedfe4887857a97939ba14401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 00:45:07.228855 31713 leader_election.cc:302] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 433319dcedfe4887857a97939ba14401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 433319dcedfe4887857a97939ba14401, 658b70fbcba84147b8c6bb3dee887e7d; no voters:
I20260810 00:45:07.228926 32177 log.cc:1196] T 1ca64df3ab584cdea699ba758e88a0f7 P 433319dcedfe4887857a97939ba14401: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-1/wal/wals/1ca64df3ab584cdea699ba758e88a0f7
I20260810 00:45:07.229431 32166 raft_consensus.cc:2781] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 433319dcedfe4887857a97939ba14401 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 00:45:07.229568 32166 raft_consensus.cc:482] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 433319dcedfe4887857a97939ba14401 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 00:45:07.229676 32166 raft_consensus.cc:3037] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 433319dcedfe4887857a97939ba14401 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:07.230476 32177 ts_tablet_manager.cc:1939] T 1ca64df3ab584cdea699ba758e88a0f7 P 433319dcedfe4887857a97939ba14401: Deleting consensus metadata
I20260810 00:45:07.232357 31353 catalog_manager.cc:4841] TS 433319dcedfe4887857a97939ba14401 (127.28.6.2:46687): tablet 1ca64df3ab584cdea699ba758e88a0f7 (table test-table [id=2213e988d3f94fd88ec9c34dd970eda1]) successfully deleted
I20260810 00:45:07.233393 32166 raft_consensus.cc:504] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 433319dcedfe4887857a97939ba14401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:07.234021 32166 leader_election.cc:288] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 433319dcedfe4887857a97939ba14401 [CANDIDATE]: Term 1 election: Requested vote from peers 875b3956c2304204aff06dbd87048d09 (127.28.6.1:45757), 658b70fbcba84147b8c6bb3dee887e7d (127.28.6.3:41205)
I20260810 00:45:07.234496 31612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c843aa5cb24c8d9eb824bc2dee05b0" candidate_uuid: "433319dcedfe4887857a97939ba14401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "875b3956c2304204aff06dbd87048d09"
I20260810 00:45:07.234596 32008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c843aa5cb24c8d9eb824bc2dee05b0" candidate_uuid: "433319dcedfe4887857a97939ba14401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "658b70fbcba84147b8c6bb3dee887e7d"
I20260810 00:45:07.234719 31612 raft_consensus.cc:3037] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 875b3956c2304204aff06dbd87048d09 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:07.234818 32008 raft_consensus.cc:3037] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 658b70fbcba84147b8c6bb3dee887e7d [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:07.237903 31612 raft_consensus.cc:2454] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 875b3956c2304204aff06dbd87048d09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 433319dcedfe4887857a97939ba14401 in term 1.
I20260810 00:45:07.237908 32008 raft_consensus.cc:2454] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 658b70fbcba84147b8c6bb3dee887e7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 433319dcedfe4887857a97939ba14401 in term 1.
I20260810 00:45:07.238440 31713 leader_election.cc:302] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 433319dcedfe4887857a97939ba14401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 433319dcedfe4887857a97939ba14401, 875b3956c2304204aff06dbd87048d09; no voters:
I20260810 00:45:07.238714 32166 raft_consensus.cc:2781] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 433319dcedfe4887857a97939ba14401 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:45:07.238936 32166 raft_consensus.cc:686] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 433319dcedfe4887857a97939ba14401 [term 1 LEADER]: Becoming Leader. State: Replica: 433319dcedfe4887857a97939ba14401, State: Running, Role: LEADER
I20260810 00:45:07.239471 32166 consensus_queue.cc:237] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 433319dcedfe4887857a97939ba14401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:07.242223 31370 catalog_manager.cc:5495] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 433319dcedfe4887857a97939ba14401 reported cstate change: term changed from 0 to 1, leader changed from <none> to 433319dcedfe4887857a97939ba14401 (127.28.6.2). New cstate: current_term: 1 leader_uuid: "433319dcedfe4887857a97939ba14401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } health_report { overall_health: UNKNOWN } } }
I20260810 00:45:07.279969 31615 raft_consensus.cc:1261] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 875b3956c2304204aff06dbd87048d09 [term 1 FOLLOWER]: Refusing update from remote peer 433319dcedfe4887857a97939ba14401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:45:07.280622 32008 raft_consensus.cc:1261] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 658b70fbcba84147b8c6bb3dee887e7d [term 1 FOLLOWER]: Refusing update from remote peer 433319dcedfe4887857a97939ba14401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:45:07.280582 32166 consensus_queue.cc:1035] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 433319dcedfe4887857a97939ba14401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:45:07.281237 32126 consensus_queue.cc:1035] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 433319dcedfe4887857a97939ba14401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:45:07.298498 32146 mvcc.cc:205] Tried to move back new op lower bound from 7316777809012232192 to 7316777808855171072. Current Snapshot: MvccSnapshot[applied={T|T < 7316777809012232192}]
W20260810 00:45:07.319195 32147 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 00:45:08.919448 31370 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:45:08.919831 31370 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:45:08.924835 31370 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7678fc95ce74a4ba1a31ad544a71001: Sending DeleteTablet for 3 replicas of tablet e8c843aa5cb24c8d9eb824bc2dee05b0
I20260810 00:45:08.925648 31787 tablet_service.cc:1514] Processing DeleteTablet for tablet e8c843aa5cb24c8d9eb824bc2dee05b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:45:08 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:48322
I20260810 00:45:08.925750 31575 tablet_service.cc:1514] Processing DeleteTablet for tablet e8c843aa5cb24c8d9eb824bc2dee05b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:45:08 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:39678
I20260810 00:45:08.925858 31975 tablet_service.cc:1514] Processing DeleteTablet for tablet e8c843aa5cb24c8d9eb824bc2dee05b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:45:08 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:36782
I20260810 00:45:08.939265 32222 tablet_replica.cc:330] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 658b70fbcba84147b8c6bb3dee887e7d: stopping tablet replica
I20260810 00:45:08.939473 32221 tablet_replica.cc:330] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 875b3956c2304204aff06dbd87048d09: stopping tablet replica
I20260810 00:45:08.939512 32220 tablet_replica.cc:330] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 433319dcedfe4887857a97939ba14401: stopping tablet replica
I20260810 00:45:08.939605 32222 raft_consensus.cc:2229] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 658b70fbcba84147b8c6bb3dee887e7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:45:08.939783 32220 raft_consensus.cc:2229] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 433319dcedfe4887857a97939ba14401 [term 1 LEADER]: Raft consensus shutting down.
I20260810 00:45:08.939807 32221 raft_consensus.cc:2229] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 875b3956c2304204aff06dbd87048d09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:45:08.939965 32222 raft_consensus.cc:2258] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 658b70fbcba84147b8c6bb3dee887e7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:45:08.940070 32221 raft_consensus.cc:2258] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 875b3956c2304204aff06dbd87048d09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:45:08.940507 32220 raft_consensus.cc:2258] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 433319dcedfe4887857a97939ba14401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:45:08.941252 32222 ts_tablet_manager.cc:1905] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 658b70fbcba84147b8c6bb3dee887e7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:45:08.941310 32221 ts_tablet_manager.cc:1905] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 875b3956c2304204aff06dbd87048d09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:45:08.941743 32220 ts_tablet_manager.cc:1905] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 433319dcedfe4887857a97939ba14401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:45:08.950126 32222 ts_tablet_manager.cc:1918] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 658b70fbcba84147b8c6bb3dee887e7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 00:45:08.950477 32222 log.cc:1196] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 658b70fbcba84147b8c6bb3dee887e7d: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-2/wal/wals/e8c843aa5cb24c8d9eb824bc2dee05b0
I20260810 00:45:08.950480 32221 ts_tablet_manager.cc:1918] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 875b3956c2304204aff06dbd87048d09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 00:45:08.950659 32220 ts_tablet_manager.cc:1918] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 433319dcedfe4887857a97939ba14401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 00:45:08.950757 32221 log.cc:1196] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 875b3956c2304204aff06dbd87048d09: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-0/wal/wals/e8c843aa5cb24c8d9eb824bc2dee05b0
I20260810 00:45:08.950940 32220 log.cc:1196] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 433319dcedfe4887857a97939ba14401: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-1/wal/wals/e8c843aa5cb24c8d9eb824bc2dee05b0
I20260810 00:45:08.951846 32221 ts_tablet_manager.cc:1939] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 875b3956c2304204aff06dbd87048d09: Deleting consensus metadata
I20260810 00:45:08.952294 32222 ts_tablet_manager.cc:1939] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 658b70fbcba84147b8c6bb3dee887e7d: Deleting consensus metadata
I20260810 00:45:08.951882 32220 ts_tablet_manager.cc:1939] T e8c843aa5cb24c8d9eb824bc2dee05b0 P 433319dcedfe4887857a97939ba14401: Deleting consensus metadata
I20260810 00:45:08.953549 31352 catalog_manager.cc:4841] TS 658b70fbcba84147b8c6bb3dee887e7d (127.28.6.3:41205): tablet e8c843aa5cb24c8d9eb824bc2dee05b0 (table test-table [id=1391b8fce3f24897afe09e32574496d7]) successfully deleted
I20260810 00:45:08.953889 31352 catalog_manager.cc:4841] TS 875b3956c2304204aff06dbd87048d09 (127.28.6.1:45757): tablet e8c843aa5cb24c8d9eb824bc2dee05b0 (table test-table [id=1391b8fce3f24897afe09e32574496d7]) successfully deleted
I20260810 00:45:08.953894 31353 catalog_manager.cc:4841] TS 433319dcedfe4887857a97939ba14401 (127.28.6.2:46687): tablet e8c843aa5cb24c8d9eb824bc2dee05b0 (table test-table [id=1391b8fce3f24897afe09e32574496d7]) successfully deleted
Aug 10 00:45:08 dist-test-slave-1nl6 krb5kdc[31306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322708, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 00:45:08 dist-test-slave-1nl6 krb5kdc[31306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322708, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:45:08.993582 32237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:08.973553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46746 (local address 127.28.6.62:41269)
0810 00:45:08.976271 (+ 2718us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:08.976276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:08.976293 (+ 17us) server_negotiation.cc:407] Connection header received
0810 00:45:08.976345 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:08.976349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:08.976402 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:08.976485 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:08.978442 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:08.979321 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:08.981059 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:08.981695 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:08.981795 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:08.987059 (+ 5264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:08.987065 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:08.987069 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:08.987129 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:08.990683 (+ 3554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:08.991486 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:08.991490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:08.991493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:08.991736 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:08.992117 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:08.992120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:08.992123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:08.992506 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:08.992718 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:08.993033 (+ 315us) server_negotiation.cc:299] Negotiation successful
0810 00:45:08.993205 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2608,"thread_start_us":2418,"threads_started":1}
I20260810 00:45:08.997639 31370 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 00:45:08.998365 31370 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 00:45:09.005648 31575 tablet_service.cc:1467] Processing CreateTablet for tablet ea393ddc74a14d4ca178c28739501679 (DEFAULT_TABLE table=test-table [id=3d5003c51fce424e8a6933d72a3274f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:09.005888 31787 tablet_service.cc:1467] Processing CreateTablet for tablet ea393ddc74a14d4ca178c28739501679 (DEFAULT_TABLE table=test-table [id=3d5003c51fce424e8a6933d72a3274f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:09.006336 31575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea393ddc74a14d4ca178c28739501679. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:09.006170 31975 tablet_service.cc:1467] Processing CreateTablet for tablet ea393ddc74a14d4ca178c28739501679 (DEFAULT_TABLE table=test-table [id=3d5003c51fce424e8a6933d72a3274f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:09.006397 31787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea393ddc74a14d4ca178c28739501679. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:09.006742 31975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea393ddc74a14d4ca178c28739501679. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:09.014950 32239 tablet_bootstrap.cc:492] T ea393ddc74a14d4ca178c28739501679 P 433319dcedfe4887857a97939ba14401: Bootstrap starting.
I20260810 00:45:09.018425 32239 tablet_bootstrap.cc:654] T ea393ddc74a14d4ca178c28739501679 P 433319dcedfe4887857a97939ba14401: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:09.019904 32240 tablet_bootstrap.cc:492] T ea393ddc74a14d4ca178c28739501679 P 658b70fbcba84147b8c6bb3dee887e7d: Bootstrap starting.
I20260810 00:45:09.022954 32242 tablet_bootstrap.cc:492] T ea393ddc74a14d4ca178c28739501679 P 875b3956c2304204aff06dbd87048d09: Bootstrap starting.
I20260810 00:45:09.023501 32240 tablet_bootstrap.cc:654] T ea393ddc74a14d4ca178c28739501679 P 658b70fbcba84147b8c6bb3dee887e7d: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:09.024168 32239 tablet_bootstrap.cc:492] T ea393ddc74a14d4ca178c28739501679 P 433319dcedfe4887857a97939ba14401: No bootstrap required, opened a new log
I20260810 00:45:09.024436 32239 ts_tablet_manager.cc:1397] T ea393ddc74a14d4ca178c28739501679 P 433319dcedfe4887857a97939ba14401: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260810 00:45:09.026103 32239 raft_consensus.cc:348] T ea393ddc74a14d4ca178c28739501679 P 433319dcedfe4887857a97939ba14401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:09.026407 32242 tablet_bootstrap.cc:654] T ea393ddc74a14d4ca178c28739501679 P 875b3956c2304204aff06dbd87048d09: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:09.026454 32239 raft_consensus.cc:374] T ea393ddc74a14d4ca178c28739501679 P 433319dcedfe4887857a97939ba14401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:09.026691 32239 raft_consensus.cc:729] T ea393ddc74a14d4ca178c28739501679 P 433319dcedfe4887857a97939ba14401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 433319dcedfe4887857a97939ba14401, State: Initialized, Role: FOLLOWER
I20260810 00:45:09.027272 32239 consensus_queue.cc:260] T ea393ddc74a14d4ca178c28739501679 P 433319dcedfe4887857a97939ba14401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:09.028146 32239 ts_tablet_manager.cc:1428] T ea393ddc74a14d4ca178c28739501679 P 433319dcedfe4887857a97939ba14401: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 00:45:09.028517 32240 tablet_bootstrap.cc:492] T ea393ddc74a14d4ca178c28739501679 P 658b70fbcba84147b8c6bb3dee887e7d: No bootstrap required, opened a new log
I20260810 00:45:09.028815 32240 ts_tablet_manager.cc:1397] T ea393ddc74a14d4ca178c28739501679 P 658b70fbcba84147b8c6bb3dee887e7d: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260810 00:45:09.030510 32240 raft_consensus.cc:348] T ea393ddc74a14d4ca178c28739501679 P 658b70fbcba84147b8c6bb3dee887e7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:09.030853 32240 raft_consensus.cc:374] T ea393ddc74a14d4ca178c28739501679 P 658b70fbcba84147b8c6bb3dee887e7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:09.031097 32240 raft_consensus.cc:729] T ea393ddc74a14d4ca178c28739501679 P 658b70fbcba84147b8c6bb3dee887e7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 658b70fbcba84147b8c6bb3dee887e7d, State: Initialized, Role: FOLLOWER
I20260810 00:45:09.031597 32242 tablet_bootstrap.cc:492] T ea393ddc74a14d4ca178c28739501679 P 875b3956c2304204aff06dbd87048d09: No bootstrap required, opened a new log
I20260810 00:45:09.031529 32240 consensus_queue.cc:260] T ea393ddc74a14d4ca178c28739501679 P 658b70fbcba84147b8c6bb3dee887e7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:09.031807 32242 ts_tablet_manager.cc:1397] T ea393ddc74a14d4ca178c28739501679 P 875b3956c2304204aff06dbd87048d09: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260810 00:45:09.032894 32240 ts_tablet_manager.cc:1428] T ea393ddc74a14d4ca178c28739501679 P 658b70fbcba84147b8c6bb3dee887e7d: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260810 00:45:09.033396 32242 raft_consensus.cc:348] T ea393ddc74a14d4ca178c28739501679 P 875b3956c2304204aff06dbd87048d09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:09.033890 32242 raft_consensus.cc:374] T ea393ddc74a14d4ca178c28739501679 P 875b3956c2304204aff06dbd87048d09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:09.034157 32242 raft_consensus.cc:729] T ea393ddc74a14d4ca178c28739501679 P 875b3956c2304204aff06dbd87048d09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 875b3956c2304204aff06dbd87048d09, State: Initialized, Role: FOLLOWER
I20260810 00:45:09.034559 32242 consensus_queue.cc:260] T ea393ddc74a14d4ca178c28739501679 P 875b3956c2304204aff06dbd87048d09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:09.036474 32242 ts_tablet_manager.cc:1428] T ea393ddc74a14d4ca178c28739501679 P 875b3956c2304204aff06dbd87048d09: Time spent starting tablet: real 0.004s user 0.002s sys 0.003s
I20260810 00:45:09.190467 32247 raft_consensus.cc:482] T ea393ddc74a14d4ca178c28739501679 P 875b3956c2304204aff06dbd87048d09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:45:09.190742 32247 raft_consensus.cc:504] T ea393ddc74a14d4ca178c28739501679 P 875b3956c2304204aff06dbd87048d09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:09.192691 32247 leader_election.cc:288] T ea393ddc74a14d4ca178c28739501679 P 875b3956c2304204aff06dbd87048d09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 433319dcedfe4887857a97939ba14401 (127.28.6.2:46687), 658b70fbcba84147b8c6bb3dee887e7d (127.28.6.3:41205)
I20260810 00:45:09.213311 32254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:09.193263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:41593 (local address 127.28.6.2:46687)
0810 00:45:09.206688 (+ 13425us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:09.206695 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:09.206721 (+ 26us) server_negotiation.cc:407] Connection header received
0810 00:45:09.206797 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:09.206802 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:09.206869 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:09.207017 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:09.207874 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:09.208847 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:09.210978 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:09.211939 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:09.212093 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:09.212294 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:09.212572 (+ 278us) server_negotiation.cc:299] Negotiation successful
0810 00:45:09.212720 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13176,"thread_start_us":12929,"threads_started":1}
I20260810 00:45:09.213442 32253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:09.192924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:46687 (local address 127.28.6.1:41593)
0810 00:45:09.205690 (+ 12766us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:09.205713 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:09.206028 (+ 315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:09.207044 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:09.207067 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:09.207092 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:09.207657 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:09.207669 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:09.209024 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:09.209028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:09.210774 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:09.210787 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:09.212167 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:09.212174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:09.212389 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:09.212414 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 00:45:09.212545 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 00:45:09.212687 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12523,"spinlock_wait_cycles":20224,"thread_start_us":13227,"threads_started":1}
I20260810 00:45:09.213788 32251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:09.193186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:52745 (local address 127.28.6.3:41205)
0810 00:45:09.205659 (+ 12473us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:09.205668 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:09.206620 (+ 952us) server_negotiation.cc:407] Connection header received
0810 00:45:09.206816 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:09.206820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:09.206885 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:09.207065 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:09.208339 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:09.209302 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:09.211334 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:09.212355 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:09.212516 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:09.212647 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:09.212986 (+ 339us) server_negotiation.cc:299] Negotiation successful
0810 00:45:09.213152 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12272,"thread_start_us":12046,"threads_started":1}
I20260810 00:45:09.214066 32249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:09.193295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:41205 (local address 127.28.6.1:52745)
0810 00:45:09.206507 (+ 13212us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:09.206535 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:09.206687 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:09.207090 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:09.207095 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:09.207125 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:09.208088 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:09.208100 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:09.209448 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:09.209455 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:09.211186 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:09.211200 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:09.212620 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:09.212629 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:09.212819 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:09.212844 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 00:45:09.212960 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 00:45:09.213081 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13071,"spinlock_wait_cycles":768,"thread_start_us":12398,"threads_started":1}
I20260810 00:45:09.214403 31822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea393ddc74a14d4ca178c28739501679" candidate_uuid: "875b3956c2304204aff06dbd87048d09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "433319dcedfe4887857a97939ba14401" is_pre_election: true
I20260810 00:45:09.214687 31822 raft_consensus.cc:2454] T ea393ddc74a14d4ca178c28739501679 P 433319dcedfe4887857a97939ba14401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 875b3956c2304204aff06dbd87048d09 in term 0.
I20260810 00:45:09.214771 32005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea393ddc74a14d4ca178c28739501679" candidate_uuid: "875b3956c2304204aff06dbd87048d09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "658b70fbcba84147b8c6bb3dee887e7d" is_pre_election: true
I20260810 00:45:09.215133 32005 raft_consensus.cc:2454] T ea393ddc74a14d4ca178c28739501679 P 658b70fbcba84147b8c6bb3dee887e7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 875b3956c2304204aff06dbd87048d09 in term 0.
I20260810 00:45:09.215371 31503 leader_election.cc:302] T ea393ddc74a14d4ca178c28739501679 P 875b3956c2304204aff06dbd87048d09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 433319dcedfe4887857a97939ba14401, 875b3956c2304204aff06dbd87048d09; no voters:
I20260810 00:45:09.215961 32247 raft_consensus.cc:2781] T ea393ddc74a14d4ca178c28739501679 P 875b3956c2304204aff06dbd87048d09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 00:45:09.216151 32247 raft_consensus.cc:482] T ea393ddc74a14d4ca178c28739501679 P 875b3956c2304204aff06dbd87048d09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 00:45:09.216298 32247 raft_consensus.cc:3037] T ea393ddc74a14d4ca178c28739501679 P 875b3956c2304204aff06dbd87048d09 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:09.219199 32247 raft_consensus.cc:504] T ea393ddc74a14d4ca178c28739501679 P 875b3956c2304204aff06dbd87048d09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:09.219964 32247 leader_election.cc:288] T ea393ddc74a14d4ca178c28739501679 P 875b3956c2304204aff06dbd87048d09 [CANDIDATE]: Term 1 election: Requested vote from peers 433319dcedfe4887857a97939ba14401 (127.28.6.2:46687), 658b70fbcba84147b8c6bb3dee887e7d (127.28.6.3:41205)
I20260810 00:45:09.220355 31822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea393ddc74a14d4ca178c28739501679" candidate_uuid: "875b3956c2304204aff06dbd87048d09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "433319dcedfe4887857a97939ba14401"
I20260810 00:45:09.220590 31822 raft_consensus.cc:3037] T ea393ddc74a14d4ca178c28739501679 P 433319dcedfe4887857a97939ba14401 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:09.220582 32005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea393ddc74a14d4ca178c28739501679" candidate_uuid: "875b3956c2304204aff06dbd87048d09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "658b70fbcba84147b8c6bb3dee887e7d"
I20260810 00:45:09.220866 32005 raft_consensus.cc:3037] T ea393ddc74a14d4ca178c28739501679 P 658b70fbcba84147b8c6bb3dee887e7d [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:09.224350 31822 raft_consensus.cc:2454] T ea393ddc74a14d4ca178c28739501679 P 433319dcedfe4887857a97939ba14401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 875b3956c2304204aff06dbd87048d09 in term 1.
I20260810 00:45:09.224588 32005 raft_consensus.cc:2454] T ea393ddc74a14d4ca178c28739501679 P 658b70fbcba84147b8c6bb3dee887e7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 875b3956c2304204aff06dbd87048d09 in term 1.
I20260810 00:45:09.224804 31503 leader_election.cc:302] T ea393ddc74a14d4ca178c28739501679 P 875b3956c2304204aff06dbd87048d09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 433319dcedfe4887857a97939ba14401, 875b3956c2304204aff06dbd87048d09; no voters:
I20260810 00:45:09.225144 32247 raft_consensus.cc:2781] T ea393ddc74a14d4ca178c28739501679 P 875b3956c2304204aff06dbd87048d09 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:45:09.225705 32247 raft_consensus.cc:686] T ea393ddc74a14d4ca178c28739501679 P 875b3956c2304204aff06dbd87048d09 [term 1 LEADER]: Becoming Leader. State: Replica: 875b3956c2304204aff06dbd87048d09, State: Running, Role: LEADER
I20260810 00:45:09.226104 32247 consensus_queue.cc:237] T ea393ddc74a14d4ca178c28739501679 P 875b3956c2304204aff06dbd87048d09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:09.229893 31370 catalog_manager.cc:5495] T ea393ddc74a14d4ca178c28739501679 P 875b3956c2304204aff06dbd87048d09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 875b3956c2304204aff06dbd87048d09 (127.28.6.1). New cstate: current_term: 1 leader_uuid: "875b3956c2304204aff06dbd87048d09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } health_report { overall_health: UNKNOWN } } }
I20260810 00:45:09.268056 32258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:09.261271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56648 (local address 127.28.6.1:45757)
0810 00:45:09.262145 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:09.262149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:09.262175 (+ 26us) server_negotiation.cc:407] Connection header received
0810 00:45:09.262254 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:09.262257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:09.262340 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:09.262524 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:45:09.263113 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:09.264014 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:09.265425 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:09.265963 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:09.266103 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:09.266614 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:45:09.267311 (+ 697us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:45:09.267526 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:09.267578 (+ 52us) server_negotiation.cc:299] Negotiation successful
0810 00:45:09.267652 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":581,"threads_started":1}
I20260810 00:45:09.286626 32251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:09.279497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:43453 (local address 127.28.6.3:41205)
0810 00:45:09.279689 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:09.279694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:09.280548 (+ 854us) server_negotiation.cc:407] Connection header received
0810 00:45:09.280807 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:09.280812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:09.280879 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:09.280979 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:45:09.281677 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:09.282441 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:09.284767 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:09.285229 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:09.285383 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:09.285873 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:45:09.285971 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:45:09.286072 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:09.286212 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 00:45:09.286290 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 00:45:09.286819 32265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:09.279574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:41205 (local address 127.28.6.1:43453)
0810 00:45:09.280438 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:09.280454 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:09.280652 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:09.280993 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:09.280998 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:09.281009 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 00:45:09.281466 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:09.281476 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:09.282613 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:09.282621 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:09.284598 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:09.284612 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:09.285455 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:09.285462 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:09.285611 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:09.285672 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 00:45:09.286077 (+ 405us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 00:45:09.286090 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 00:45:09.286173 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 00:45:09.286304 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":476,"threads_started":1}
I20260810 00:45:09.295527 32005 raft_consensus.cc:1261] T ea393ddc74a14d4ca178c28739501679 P 658b70fbcba84147b8c6bb3dee887e7d [term 1 FOLLOWER]: Refusing update from remote peer 875b3956c2304204aff06dbd87048d09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:45:09.296379 31822 raft_consensus.cc:1261] T ea393ddc74a14d4ca178c28739501679 P 433319dcedfe4887857a97939ba14401 [term 1 FOLLOWER]: Refusing update from remote peer 875b3956c2304204aff06dbd87048d09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:45:09.296229 32255 consensus_queue.cc:1035] T ea393ddc74a14d4ca178c28739501679 P 875b3956c2304204aff06dbd87048d09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:45:09.299196 32255 consensus_queue.cc:1035] T ea393ddc74a14d4ca178c28739501679 P 875b3956c2304204aff06dbd87048d09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:45:09.337087 32258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:09.319564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:43735 (local address 127.28.6.1:45757)
0810 00:45:09.319734 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:09.319738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:09.331067 (+ 11329us) server_negotiation.cc:407] Connection header received
0810 00:45:09.331250 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:09.331255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:09.331344 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:09.331437 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:45:09.332046 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:09.333337 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:09.334736 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:09.335611 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:09.335724 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:09.336130 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:45:09.336295 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:45:09.336406 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:09.336589 (+ 183us) server_negotiation.cc:299] Negotiation successful
0810 00:45:09.336685 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 00:45:09.337911 32274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:09.319432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:45757 (local address 127.28.6.3:43735)
0810 00:45:09.330953 (+ 11521us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:09.330971 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:09.331108 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:09.331449 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:09.331454 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:09.331465 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 00:45:09.331827 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:09.331842 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:09.333489 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:09.333493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:09.334594 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:09.334606 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:09.335774 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:09.335778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:09.335894 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:09.335929 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 00:45:09.336462 (+ 533us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 00:45:09.336474 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 00:45:09.337328 (+ 854us) client_negotiation.cc:240] Negotiation successful
0810 00:45:09.337472 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11303,"thread_start_us":11131,"threads_started":1}
I20260810 00:45:09.376521 31372 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:45:09.376857 31372 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:45:09.380466 31372 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7678fc95ce74a4ba1a31ad544a71001: Sending DeleteTablet for 3 replicas of tablet ea393ddc74a14d4ca178c28739501679
I20260810 00:45:09.381518 31575 tablet_service.cc:1514] Processing DeleteTablet for tablet ea393ddc74a14d4ca178c28739501679 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:45:09 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:39678
I20260810 00:45:09.381640 31975 tablet_service.cc:1514] Processing DeleteTablet for tablet ea393ddc74a14d4ca178c28739501679 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:45:09 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:36782
I20260810 00:45:09.381858 32221 tablet_replica.cc:330] T ea393ddc74a14d4ca178c28739501679 P 875b3956c2304204aff06dbd87048d09: stopping tablet replica
I20260810 00:45:09.381970 32222 tablet_replica.cc:330] T ea393ddc74a14d4ca178c28739501679 P 658b70fbcba84147b8c6bb3dee887e7d: stopping tablet replica
I20260810 00:45:09.382097 32221 raft_consensus.cc:2229] T ea393ddc74a14d4ca178c28739501679 P 875b3956c2304204aff06dbd87048d09 [term 1 LEADER]: Raft consensus shutting down.
I20260810 00:45:09.382395 32222 raft_consensus.cc:2229] T ea393ddc74a14d4ca178c28739501679 P 658b70fbcba84147b8c6bb3dee887e7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:45:09.382674 32222 raft_consensus.cc:2258] T ea393ddc74a14d4ca178c28739501679 P 658b70fbcba84147b8c6bb3dee887e7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:45:09.382962 31372 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 00:45:09.383893 31372 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 00:45:09.384984 32221 raft_consensus.cc:2258] T ea393ddc74a14d4ca178c28739501679 P 875b3956c2304204aff06dbd87048d09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:45:09.386708 32222 ts_tablet_manager.cc:1905] T ea393ddc74a14d4ca178c28739501679 P 658b70fbcba84147b8c6bb3dee887e7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:45:09.389268 31787 tablet_service.cc:1514] Processing DeleteTablet for tablet ea393ddc74a14d4ca178c28739501679 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:45:09 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:48322
I20260810 00:45:09.389707 32220 tablet_replica.cc:330] T ea393ddc74a14d4ca178c28739501679 P 433319dcedfe4887857a97939ba14401: stopping tablet replica
I20260810 00:45:09.389936 32220 raft_consensus.cc:2229] T ea393ddc74a14d4ca178c28739501679 P 433319dcedfe4887857a97939ba14401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:45:09.390187 32220 raft_consensus.cc:2258] T ea393ddc74a14d4ca178c28739501679 P 433319dcedfe4887857a97939ba14401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:45:09.394008 32222 ts_tablet_manager.cc:1918] T ea393ddc74a14d4ca178c28739501679 P 658b70fbcba84147b8c6bb3dee887e7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 00:45:09.394266 32222 log.cc:1196] T ea393ddc74a14d4ca178c28739501679 P 658b70fbcba84147b8c6bb3dee887e7d: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-2/wal/wals/ea393ddc74a14d4ca178c28739501679
I20260810 00:45:09.394860 32222 ts_tablet_manager.cc:1939] T ea393ddc74a14d4ca178c28739501679 P 658b70fbcba84147b8c6bb3dee887e7d: Deleting consensus metadata
I20260810 00:45:09.395787 32220 ts_tablet_manager.cc:1905] T ea393ddc74a14d4ca178c28739501679 P 433319dcedfe4887857a97939ba14401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:45:09.395942 31352 catalog_manager.cc:4841] TS 658b70fbcba84147b8c6bb3dee887e7d (127.28.6.3:41205): tablet ea393ddc74a14d4ca178c28739501679 (table test-table [id=3d5003c51fce424e8a6933d72a3274f3]) successfully deleted
I20260810 00:45:09.399080 32221 ts_tablet_manager.cc:1905] T ea393ddc74a14d4ca178c28739501679 P 875b3956c2304204aff06dbd87048d09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:45:09.404989 31787 tablet_service.cc:1467] Processing CreateTablet for tablet 45899cd729134126ae5edb88724469f7 (DEFAULT_TABLE table=test-table [id=6a533be8b2174b8c8e886fe7c389f507]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:09.405611 31787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45899cd729134126ae5edb88724469f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:09.407063 32221 ts_tablet_manager.cc:1918] T ea393ddc74a14d4ca178c28739501679 P 875b3956c2304204aff06dbd87048d09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 00:45:09.407459 32221 log.cc:1196] T ea393ddc74a14d4ca178c28739501679 P 875b3956c2304204aff06dbd87048d09: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-0/wal/wals/ea393ddc74a14d4ca178c28739501679
I20260810 00:45:09.408268 32221 ts_tablet_manager.cc:1939] T ea393ddc74a14d4ca178c28739501679 P 875b3956c2304204aff06dbd87048d09: Deleting consensus metadata
I20260810 00:45:09.411878 31575 tablet_service.cc:1467] Processing CreateTablet for tablet 45899cd729134126ae5edb88724469f7 (DEFAULT_TABLE table=test-table [id=6a533be8b2174b8c8e886fe7c389f507]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:09.412212 31975 tablet_service.cc:1467] Processing CreateTablet for tablet 45899cd729134126ae5edb88724469f7 (DEFAULT_TABLE table=test-table [id=6a533be8b2174b8c8e886fe7c389f507]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:09.412322 31352 catalog_manager.cc:4841] TS 875b3956c2304204aff06dbd87048d09 (127.28.6.1:45757): tablet ea393ddc74a14d4ca178c28739501679 (table test-table [id=3d5003c51fce424e8a6933d72a3274f3]) successfully deleted
I20260810 00:45:09.412474 31575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45899cd729134126ae5edb88724469f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:09.412860 31975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45899cd729134126ae5edb88724469f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:09.417312 32220 ts_tablet_manager.cc:1918] T ea393ddc74a14d4ca178c28739501679 P 433319dcedfe4887857a97939ba14401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 00:45:09.417634 32220 log.cc:1196] T ea393ddc74a14d4ca178c28739501679 P 433319dcedfe4887857a97939ba14401: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-1/wal/wals/ea393ddc74a14d4ca178c28739501679
I20260810 00:45:09.419528 32239 tablet_bootstrap.cc:492] T 45899cd729134126ae5edb88724469f7 P 433319dcedfe4887857a97939ba14401: Bootstrap starting.
I20260810 00:45:09.419687 32220 ts_tablet_manager.cc:1939] T ea393ddc74a14d4ca178c28739501679 P 433319dcedfe4887857a97939ba14401: Deleting consensus metadata
I20260810 00:45:09.421677 32240 tablet_bootstrap.cc:492] T 45899cd729134126ae5edb88724469f7 P 658b70fbcba84147b8c6bb3dee887e7d: Bootstrap starting.
I20260810 00:45:09.422816 32239 tablet_bootstrap.cc:654] T 45899cd729134126ae5edb88724469f7 P 433319dcedfe4887857a97939ba14401: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:09.424328 32240 tablet_bootstrap.cc:654] T 45899cd729134126ae5edb88724469f7 P 658b70fbcba84147b8c6bb3dee887e7d: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:09.424451 31353 catalog_manager.cc:4841] TS 433319dcedfe4887857a97939ba14401 (127.28.6.2:46687): tablet ea393ddc74a14d4ca178c28739501679 (table test-table [id=3d5003c51fce424e8a6933d72a3274f3]) successfully deleted
I20260810 00:45:09.428399 32242 tablet_bootstrap.cc:492] T 45899cd729134126ae5edb88724469f7 P 875b3956c2304204aff06dbd87048d09: Bootstrap starting.
I20260810 00:45:09.431491 32242 tablet_bootstrap.cc:654] T 45899cd729134126ae5edb88724469f7 P 875b3956c2304204aff06dbd87048d09: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:09.431885 32239 tablet_bootstrap.cc:492] T 45899cd729134126ae5edb88724469f7 P 433319dcedfe4887857a97939ba14401: No bootstrap required, opened a new log
I20260810 00:45:09.432206 32239 ts_tablet_manager.cc:1397] T 45899cd729134126ae5edb88724469f7 P 433319dcedfe4887857a97939ba14401: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.006s
I20260810 00:45:09.433293 32239 raft_consensus.cc:348] T 45899cd729134126ae5edb88724469f7 P 433319dcedfe4887857a97939ba14401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:09.433699 32239 raft_consensus.cc:374] T 45899cd729134126ae5edb88724469f7 P 433319dcedfe4887857a97939ba14401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:09.433878 32239 raft_consensus.cc:729] T 45899cd729134126ae5edb88724469f7 P 433319dcedfe4887857a97939ba14401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 433319dcedfe4887857a97939ba14401, State: Initialized, Role: FOLLOWER
I20260810 00:45:09.434294 32239 consensus_queue.cc:260] T 45899cd729134126ae5edb88724469f7 P 433319dcedfe4887857a97939ba14401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:09.435254 32239 ts_tablet_manager.cc:1428] T 45899cd729134126ae5edb88724469f7 P 433319dcedfe4887857a97939ba14401: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260810 00:45:09.445341 32240 tablet_bootstrap.cc:492] T 45899cd729134126ae5edb88724469f7 P 658b70fbcba84147b8c6bb3dee887e7d: No bootstrap required, opened a new log
I20260810 00:45:09.445549 32240 ts_tablet_manager.cc:1397] T 45899cd729134126ae5edb88724469f7 P 658b70fbcba84147b8c6bb3dee887e7d: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.008s
I20260810 00:45:09.446626 32240 raft_consensus.cc:348] T 45899cd729134126ae5edb88724469f7 P 658b70fbcba84147b8c6bb3dee887e7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:09.447057 32240 raft_consensus.cc:374] T 45899cd729134126ae5edb88724469f7 P 658b70fbcba84147b8c6bb3dee887e7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:09.447256 32240 raft_consensus.cc:729] T 45899cd729134126ae5edb88724469f7 P 658b70fbcba84147b8c6bb3dee887e7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 658b70fbcba84147b8c6bb3dee887e7d, State: Initialized, Role: FOLLOWER
I20260810 00:45:09.447615 32240 consensus_queue.cc:260] T 45899cd729134126ae5edb88724469f7 P 658b70fbcba84147b8c6bb3dee887e7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:09.448545 32240 ts_tablet_manager.cc:1428] T 45899cd729134126ae5edb88724469f7 P 658b70fbcba84147b8c6bb3dee887e7d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 00:45:09.448969 32242 tablet_bootstrap.cc:492] T 45899cd729134126ae5edb88724469f7 P 875b3956c2304204aff06dbd87048d09: No bootstrap required, opened a new log
I20260810 00:45:09.449317 32242 ts_tablet_manager.cc:1397] T 45899cd729134126ae5edb88724469f7 P 875b3956c2304204aff06dbd87048d09: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.004s
I20260810 00:45:09.450337 32242 raft_consensus.cc:348] T 45899cd729134126ae5edb88724469f7 P 875b3956c2304204aff06dbd87048d09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:09.450654 32242 raft_consensus.cc:374] T 45899cd729134126ae5edb88724469f7 P 875b3956c2304204aff06dbd87048d09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:09.450834 32242 raft_consensus.cc:729] T 45899cd729134126ae5edb88724469f7 P 875b3956c2304204aff06dbd87048d09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 875b3956c2304204aff06dbd87048d09, State: Initialized, Role: FOLLOWER
I20260810 00:45:09.451319 32242 consensus_queue.cc:260] T 45899cd729134126ae5edb88724469f7 P 875b3956c2304204aff06dbd87048d09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:09.451990 32242 ts_tablet_manager.cc:1428] T 45899cd729134126ae5edb88724469f7 P 875b3956c2304204aff06dbd87048d09: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 00:45:09.627446 32192 raft_consensus.cc:482] T 45899cd729134126ae5edb88724469f7 P 658b70fbcba84147b8c6bb3dee887e7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:45:09.627703 32192 raft_consensus.cc:504] T 45899cd729134126ae5edb88724469f7 P 658b70fbcba84147b8c6bb3dee887e7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:09.628525 32192 leader_election.cc:288] T 45899cd729134126ae5edb88724469f7 P 658b70fbcba84147b8c6bb3dee887e7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 875b3956c2304204aff06dbd87048d09 (127.28.6.1:45757), 433319dcedfe4887857a97939ba14401 (127.28.6.2:46687)
I20260810 00:45:09.629266 31612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45899cd729134126ae5edb88724469f7" candidate_uuid: "658b70fbcba84147b8c6bb3dee887e7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "875b3956c2304204aff06dbd87048d09" is_pre_election: true
I20260810 00:45:09.629272 31822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45899cd729134126ae5edb88724469f7" candidate_uuid: "658b70fbcba84147b8c6bb3dee887e7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "433319dcedfe4887857a97939ba14401" is_pre_election: true
I20260810 00:45:09.629633 31822 raft_consensus.cc:2454] T 45899cd729134126ae5edb88724469f7 P 433319dcedfe4887857a97939ba14401 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 658b70fbcba84147b8c6bb3dee887e7d in term 0.
I20260810 00:45:09.629663 31612 raft_consensus.cc:2454] T 45899cd729134126ae5edb88724469f7 P 875b3956c2304204aff06dbd87048d09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 658b70fbcba84147b8c6bb3dee887e7d in term 0.
I20260810 00:45:09.630276 31910 leader_election.cc:302] T 45899cd729134126ae5edb88724469f7 P 658b70fbcba84147b8c6bb3dee887e7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 433319dcedfe4887857a97939ba14401, 658b70fbcba84147b8c6bb3dee887e7d, 875b3956c2304204aff06dbd87048d09; no voters:
I20260810 00:45:09.630731 32192 raft_consensus.cc:2781] T 45899cd729134126ae5edb88724469f7 P 658b70fbcba84147b8c6bb3dee887e7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 00:45:09.630973 32192 raft_consensus.cc:482] T 45899cd729134126ae5edb88724469f7 P 658b70fbcba84147b8c6bb3dee887e7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 00:45:09.631131 32192 raft_consensus.cc:3037] T 45899cd729134126ae5edb88724469f7 P 658b70fbcba84147b8c6bb3dee887e7d [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:09.633591 32192 raft_consensus.cc:504] T 45899cd729134126ae5edb88724469f7 P 658b70fbcba84147b8c6bb3dee887e7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:09.634461 32192 leader_election.cc:288] T 45899cd729134126ae5edb88724469f7 P 658b70fbcba84147b8c6bb3dee887e7d [CANDIDATE]: Term 1 election: Requested vote from peers 875b3956c2304204aff06dbd87048d09 (127.28.6.1:45757), 433319dcedfe4887857a97939ba14401 (127.28.6.2:46687)
I20260810 00:45:09.635047 31822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45899cd729134126ae5edb88724469f7" candidate_uuid: "658b70fbcba84147b8c6bb3dee887e7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "433319dcedfe4887857a97939ba14401"
I20260810 00:45:09.635035 31612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45899cd729134126ae5edb88724469f7" candidate_uuid: "658b70fbcba84147b8c6bb3dee887e7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "875b3956c2304204aff06dbd87048d09"
I20260810 00:45:09.635337 31822 raft_consensus.cc:3037] T 45899cd729134126ae5edb88724469f7 P 433319dcedfe4887857a97939ba14401 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:09.635372 31612 raft_consensus.cc:3037] T 45899cd729134126ae5edb88724469f7 P 875b3956c2304204aff06dbd87048d09 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:09.638281 31822 raft_consensus.cc:2454] T 45899cd729134126ae5edb88724469f7 P 433319dcedfe4887857a97939ba14401 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 658b70fbcba84147b8c6bb3dee887e7d in term 1.
I20260810 00:45:09.638751 31910 leader_election.cc:302] T 45899cd729134126ae5edb88724469f7 P 658b70fbcba84147b8c6bb3dee887e7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 433319dcedfe4887857a97939ba14401, 658b70fbcba84147b8c6bb3dee887e7d; no voters:
I20260810 00:45:09.639002 31612 raft_consensus.cc:2454] T 45899cd729134126ae5edb88724469f7 P 875b3956c2304204aff06dbd87048d09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 658b70fbcba84147b8c6bb3dee887e7d in term 1.
I20260810 00:45:09.639083 32192 raft_consensus.cc:2781] T 45899cd729134126ae5edb88724469f7 P 658b70fbcba84147b8c6bb3dee887e7d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:45:09.639349 32192 raft_consensus.cc:686] T 45899cd729134126ae5edb88724469f7 P 658b70fbcba84147b8c6bb3dee887e7d [term 1 LEADER]: Becoming Leader. State: Replica: 658b70fbcba84147b8c6bb3dee887e7d, State: Running, Role: LEADER
I20260810 00:45:09.639724 32192 consensus_queue.cc:237] T 45899cd729134126ae5edb88724469f7 P 658b70fbcba84147b8c6bb3dee887e7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } }
I20260810 00:45:09.643530 31370 catalog_manager.cc:5495] T 45899cd729134126ae5edb88724469f7 P 658b70fbcba84147b8c6bb3dee887e7d reported cstate change: term changed from 0 to 1, leader changed from <none> to 658b70fbcba84147b8c6bb3dee887e7d (127.28.6.3). New cstate: current_term: 1 leader_uuid: "658b70fbcba84147b8c6bb3dee887e7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "658b70fbcba84147b8c6bb3dee887e7d" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41205 } health_report { overall_health: HEALTHY } } }
I20260810 00:45:09.675078 32251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:09.669163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42454 (local address 127.28.6.3:41205)
0810 00:45:09.669353 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:09.669360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:09.669429 (+ 69us) server_negotiation.cc:407] Connection header received
0810 00:45:09.669699 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:09.669705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:09.669779 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:09.669875 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:45:09.670296 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:09.671263 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:09.672965 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:09.673468 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:09.673620 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:09.674178 (+ 558us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:45:09.674283 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:45:09.674385 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:09.674566 (+ 181us) server_negotiation.cc:299] Negotiation successful
0810 00:45:09.674646 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 00:45:09.692883 32251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:09.686455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:48255 (local address 127.28.6.3:41205)
0810 00:45:09.686613 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:09.686617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:09.686632 (+ 15us) server_negotiation.cc:407] Connection header received
0810 00:45:09.686687 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:09.686690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:09.686744 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:09.686815 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:45:09.687462 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:09.688486 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:09.690704 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:09.691363 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:09.691462 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:09.692096 (+ 634us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:45:09.692235 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:45:09.692306 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:09.692521 (+ 215us) server_negotiation.cc:299] Negotiation successful
0810 00:45:09.692580 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 00:45:09.693182 32274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:09.684258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:41205 (local address 127.28.6.3:48255)
0810 00:45:09.686137 (+ 1879us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:09.686158 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:09.686270 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:09.686873 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:09.686879 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:09.686893 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 00:45:09.687281 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:09.687305 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:09.688622 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:09.688628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:09.690545 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:09.690560 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:09.691528 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:09.691536 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:09.691761 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:09.691820 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 00:45:09.692348 (+ 528us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 00:45:09.692362 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 00:45:09.692488 (+ 126us) client_negotiation.cc:240] Negotiation successful
0810 00:45:09.692630 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1725}
I20260810 00:45:09.703536 31822 raft_consensus.cc:1261] T 45899cd729134126ae5edb88724469f7 P 433319dcedfe4887857a97939ba14401 [term 1 FOLLOWER]: Refusing update from remote peer 658b70fbcba84147b8c6bb3dee887e7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:45:09.704166 32141 consensus_queue.cc:1035] T 45899cd729134126ae5edb88724469f7 P 658b70fbcba84147b8c6bb3dee887e7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "433319dcedfe4887857a97939ba14401" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:45:09.709522 31612 raft_consensus.cc:1261] T 45899cd729134126ae5edb88724469f7 P 875b3956c2304204aff06dbd87048d09 [term 1 FOLLOWER]: Refusing update from remote peer 658b70fbcba84147b8c6bb3dee887e7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:45:09.710253 32192 consensus_queue.cc:1035] T 45899cd729134126ae5edb88724469f7 P 658b70fbcba84147b8c6bb3dee887e7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "875b3956c2304204aff06dbd87048d09" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 45757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 00:45:09.734688 32190 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 00:45:11.336546 31372 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:45:11.336817 31372 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:45:11.342144 31372 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7678fc95ce74a4ba1a31ad544a71001: Sending DeleteTablet for 3 replicas of tablet 45899cd729134126ae5edb88724469f7
I20260810 00:45:11.343149 31787 tablet_service.cc:1514] Processing DeleteTablet for tablet 45899cd729134126ae5edb88724469f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:45:11 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:48322
I20260810 00:45:11.343186 31575 tablet_service.cc:1514] Processing DeleteTablet for tablet 45899cd729134126ae5edb88724469f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:45:11 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:39678
I20260810 00:45:11.343595 31975 tablet_service.cc:1514] Processing DeleteTablet for tablet 45899cd729134126ae5edb88724469f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:45:11 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:36782
I20260810 00:45:11.343875 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 31476
I20260810 00:45:11.345378 32317 tablet_replica.cc:330] T 45899cd729134126ae5edb88724469f7 P 875b3956c2304204aff06dbd87048d09: stopping tablet replica
I20260810 00:45:11.345724 32317 raft_consensus.cc:2229] T 45899cd729134126ae5edb88724469f7 P 875b3956c2304204aff06dbd87048d09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:45:11.346240 32318 tablet_replica.cc:330] T 45899cd729134126ae5edb88724469f7 P 658b70fbcba84147b8c6bb3dee887e7d: stopping tablet replica
I20260810 00:45:11.346303 32317 raft_consensus.cc:2258] T 45899cd729134126ae5edb88724469f7 P 875b3956c2304204aff06dbd87048d09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:45:11.346575 32318 raft_consensus.cc:2229] T 45899cd729134126ae5edb88724469f7 P 658b70fbcba84147b8c6bb3dee887e7d [term 1 LEADER]: Raft consensus shutting down.
I20260810 00:45:11.348459 32318 raft_consensus.cc:2258] T 45899cd729134126ae5edb88724469f7 P 658b70fbcba84147b8c6bb3dee887e7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:45:11.350282 32318 ts_tablet_manager.cc:1905] T 45899cd729134126ae5edb88724469f7 P 658b70fbcba84147b8c6bb3dee887e7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:45:11.355816 32317 ts_tablet_manager.cc:1905] T 45899cd729134126ae5edb88724469f7 P 875b3956c2304204aff06dbd87048d09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:45:11.361259 32318 ts_tablet_manager.cc:1918] T 45899cd729134126ae5edb88724469f7 P 658b70fbcba84147b8c6bb3dee887e7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 00:45:11.361548 32318 log.cc:1196] T 45899cd729134126ae5edb88724469f7 P 658b70fbcba84147b8c6bb3dee887e7d: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-2/wal/wals/45899cd729134126ae5edb88724469f7
I20260810 00:45:11.362151 32318 ts_tablet_manager.cc:1939] T 45899cd729134126ae5edb88724469f7 P 658b70fbcba84147b8c6bb3dee887e7d: Deleting consensus metadata
I20260810 00:45:11.362201 32316 tablet_replica.cc:330] T 45899cd729134126ae5edb88724469f7 P 433319dcedfe4887857a97939ba14401: stopping tablet replica
I20260810 00:45:11.362613 32316 raft_consensus.cc:2229] T 45899cd729134126ae5edb88724469f7 P 433319dcedfe4887857a97939ba14401 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:45:11.363168 32316 raft_consensus.cc:2258] T 45899cd729134126ae5edb88724469f7 P 433319dcedfe4887857a97939ba14401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:45:11.363281 31352 catalog_manager.cc:4841] TS 658b70fbcba84147b8c6bb3dee887e7d (127.28.6.3:41205): tablet 45899cd729134126ae5edb88724469f7 (table test-table [id=6a533be8b2174b8c8e886fe7c389f507]) successfully deleted
I20260810 00:45:11.364284 32316 ts_tablet_manager.cc:1905] T 45899cd729134126ae5edb88724469f7 P 433319dcedfe4887857a97939ba14401: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:45:11.365548 32317 ts_tablet_manager.cc:1918] T 45899cd729134126ae5edb88724469f7 P 875b3956c2304204aff06dbd87048d09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 00:45:11.365826 32317 log.cc:1196] T 45899cd729134126ae5edb88724469f7 P 875b3956c2304204aff06dbd87048d09: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-0/wal/wals/45899cd729134126ae5edb88724469f7
I20260810 00:45:11.366938 32317 ts_tablet_manager.cc:1939] T 45899cd729134126ae5edb88724469f7 P 875b3956c2304204aff06dbd87048d09: Deleting consensus metadata
I20260810 00:45:11.368149 31352 catalog_manager.cc:4841] TS 875b3956c2304204aff06dbd87048d09 (127.28.6.1:45757): tablet 45899cd729134126ae5edb88724469f7 (table test-table [id=6a533be8b2174b8c8e886fe7c389f507]) successfully deleted
I20260810 00:45:11.371316 32316 ts_tablet_manager.cc:1918] T 45899cd729134126ae5edb88724469f7 P 433319dcedfe4887857a97939ba14401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 00:45:11.371583 32316 log.cc:1196] T 45899cd729134126ae5edb88724469f7 P 433319dcedfe4887857a97939ba14401: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786322687226232-28696-0/minicluster-data/ts-1/wal/wals/45899cd729134126ae5edb88724469f7
I20260810 00:45:11.372251 32316 ts_tablet_manager.cc:1939] T 45899cd729134126ae5edb88724469f7 P 433319dcedfe4887857a97939ba14401: Deleting consensus metadata
I20260810 00:45:11.373203 31353 catalog_manager.cc:4841] TS 433319dcedfe4887857a97939ba14401 (127.28.6.2:46687): tablet 45899cd729134126ae5edb88724469f7 (table test-table [id=6a533be8b2174b8c8e886fe7c389f507]) successfully deleted
Aug 10 00:45:11 dist-test-slave-1nl6 krb5kdc[31306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322708, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.6.1@KRBTEST.COM
I20260810 00:45:11.386502 32319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:11.347596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56664 (local address 127.28.6.1:45757)
0810 00:45:11.363509 (+ 15913us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:11.363515 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:11.363541 (+ 26us) server_negotiation.cc:407] Connection header received
0810 00:45:11.363598 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:11.363602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:11.363670 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:11.363820 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:11.365879 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:11.367737 (+ 1858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:11.369838 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:11.370376 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:11.370483 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:11.380058 (+ 9575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:11.380073 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:11.380090 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:11.380120 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:11.383742 (+ 3622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:11.384432 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:11.384439 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:11.384441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:11.384626 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:11.384953 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:11.384956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:11.384958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:11.385369 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:11.385603 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:11.385847 (+ 244us) server_negotiation.cc:299] Negotiation successful
0810 00:45:11.386022 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15770,"thread_start_us":15555,"threads_started":1}
W20260810 00:45:11.387445 31674 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56664
W20260810 00:45:11.388147 28696 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 31476: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 00:45:11.388404 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 31476
I20260810 00:45:11.424150 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 31686
Aug 10 00:45:11 dist-test-slave-1nl6 krb5kdc[31306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322708, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.6.2@KRBTEST.COM
I20260810 00:45:11.441929 32323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:11.424915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46870 (local address 127.28.6.2:46687)
0810 00:45:11.425450 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:11.425455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:11.425471 (+ 16us) server_negotiation.cc:407] Connection header received
0810 00:45:11.425525 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:11.425529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:11.425586 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:11.425676 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:11.427609 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:11.428334 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:11.429908 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:11.430347 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:11.430443 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:11.436196 (+ 5753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:11.436209 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:11.436229 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:11.436265 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:11.439395 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:11.440030 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:11.440038 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:11.440040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:11.440172 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:11.440498 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:11.440501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:11.440503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:11.440833 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:11.441093 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:11.441425 (+ 332us) server_negotiation.cc:299] Negotiation successful
0810 00:45:11.441656 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":304,"threads_started":1}
W20260810 00:45:11.442806 31874 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46870
W20260810 00:45:11.443421 28696 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 31686: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 00:45:11.443645 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 31686
I20260810 00:45:11.474942 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 31884
Aug 10 00:45:11 dist-test-slave-1nl6 krb5kdc[31306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322708, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.6.3@KRBTEST.COM
I20260810 00:45:11.490449 32324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:11.475882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42470 (local address 127.28.6.3:41205)
0810 00:45:11.476609 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:11.476614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:11.476629 (+ 15us) server_negotiation.cc:407] Connection header received
0810 00:45:11.476683 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:11.476687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:11.476763 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:11.476854 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:11.478283 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:11.479129 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:11.480487 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:11.480912 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:11.481003 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:11.485071 (+ 4068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:11.485086 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:11.485104 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:11.485135 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:11.488198 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:11.488800 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:11.488807 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:11.488810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:11.488936 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:11.489260 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:11.489264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:11.489266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:11.489593 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:11.489818 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:11.490014 (+ 196us) server_negotiation.cc:299] Negotiation successful
0810 00:45:11.490173 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":442,"threads_started":1}
W20260810 00:45:11.491319 32069 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42470
W20260810 00:45:11.491905 28696 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 31884: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 00:45:11.492072 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 31884
I20260810 00:45:11.524231 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 31324
I20260810 00:45:11.525432 31432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:45:11.645255 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 31324
2026-08-10T00:45:11Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9616 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:45:11 dist-test-slave-1nl6 krb5kdc[32328](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:45:11 dist-test-slave-1nl6 krb5kdc[32328](info): set up 2 sockets
Aug 10 00:45:11 dist-test-slave-1nl6 krb5kdc[32328](info): commencing operation
krb5kdc: starting...
W20260810 00:45:12.948550 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.250s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:45:13 dist-test-slave-1nl6 krb5kdc[32328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:45:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:45:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.6.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
I20260810 00:45:13.146173 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:33847
--webserver_interface=127.28.6.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:37131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:33847
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:45:13.470069 32343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:13.470379 32343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:13.470516 32343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:13.477504 32343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:45:13.477595 32343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:13.477640 32343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:13.477710 32343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:45:13.477747 32343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:45:13.487535 32343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:37131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:33847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:33847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:13.488708 32343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:13.489987 32343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:13.501969 32350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:13.501924 32351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:13.501996 32353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:13.502846 32343 server_base.cc:1034] running on GCE node
I20260810 00:45:13.503608 32343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:13.504891 32343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:13.506104 32343 hybrid_clock.cc:648] HybridClock initialized: now 1786322713506054 us; error 39 us; skew 500 ppm
I20260810 00:45:13.506481 32343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:45:13 dist-test-slave-1nl6 krb5kdc[32328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322713, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:13.509843 32343 init.cc:377] Logged in from keytab as kudu/127.28.6.62@KRBTEST.COM (short username kudu)
I20260810 00:45:13.511358 32343 webserver.cc:460] Webserver started at http://127.28.6.62:32981/ using document root <none> and password file <none>
I20260810 00:45:13.512028 32343 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:13.512185 32343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:13.512513 32343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:13.515483 32343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "a4393557b78b4243971110a7fb5f330f"
format_stamp: "Formatted at 2026-08-10 00:45:13 on dist-test-slave-1nl6"
server_key: "8b8b4b55d82439fe4036945d927b6977"
server_key_iv: "1a96242433ddd0e26ae404d121a5b263"
server_key_version: "encryptionkey@0"
I20260810 00:45:13.516264 32343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "a4393557b78b4243971110a7fb5f330f"
format_stamp: "Formatted at 2026-08-10 00:45:13 on dist-test-slave-1nl6"
server_key: "8b8b4b55d82439fe4036945d927b6977"
server_key_iv: "1a96242433ddd0e26ae404d121a5b263"
server_key_version: "encryptionkey@0"
I20260810 00:45:13.521298 32343 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260810 00:45:13.524746 32363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:13.526042 32343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 00:45:13.526244 32343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "a4393557b78b4243971110a7fb5f330f"
format_stamp: "Formatted at 2026-08-10 00:45:13 on dist-test-slave-1nl6"
server_key: "8b8b4b55d82439fe4036945d927b6977"
server_key_iv: "1a96242433ddd0e26ae404d121a5b263"
server_key_version: "encryptionkey@0"
I20260810 00:45:13.526455 32343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:13.539817 32343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:13.543960 32343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:13.544260 32343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:13.562048 32343 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:33847
I20260810 00:45:13.562081 32440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:33847 every 8 connection(s)
I20260810 00:45:13.563429 32343 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:45:13.567011 32441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:13.572238 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 32343
I20260810 00:45:13.572481 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:45:13.573110 28696 external_mini_cluster.cc:1428] Setting key a1a1617ff20e13d46a1cbe77b851435d
I20260810 00:45:13.577807 32441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4393557b78b4243971110a7fb5f330f: Bootstrap starting.
I20260810 00:45:13.581422 32441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4393557b78b4243971110a7fb5f330f: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:13.582830 32441 log.cc:824] T 00000000000000000000000000000000 P a4393557b78b4243971110a7fb5f330f: Log is configured to *not* fsync() on all Append() calls
I20260810 00:45:13.586006 32441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4393557b78b4243971110a7fb5f330f: No bootstrap required, opened a new log
Aug 10 00:45:13 dist-test-slave-1nl6 krb5kdc[32328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:45:13.596477 32441 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4393557b78b4243971110a7fb5f330f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4393557b78b4243971110a7fb5f330f" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 33847 } }
I20260810 00:45:13.596879 32441 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4393557b78b4243971110a7fb5f330f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:13.597021 32441 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4393557b78b4243971110a7fb5f330f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4393557b78b4243971110a7fb5f330f, State: Initialized, Role: FOLLOWER
I20260810 00:45:13.597748 32445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:13.575151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39812 (local address 127.28.6.62:33847)
0810 00:45:13.575973 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:13.575983 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:13.576025 (+ 42us) server_negotiation.cc:407] Connection header received
0810 00:45:13.577775 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:13.577801 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:13.578199 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:13.578507 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:13.580402 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:13.581687 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:13.583666 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:13.584496 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:13.584621 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:13.589371 (+ 4750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:13.589383 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:13.589406 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:13.589449 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:13.593508 (+ 4059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:13.594237 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:13.594245 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:13.594254 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:13.594473 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:13.594828 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:13.594835 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:13.594839 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:13.595431 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:13.595667 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:13.596100 (+ 433us) server_negotiation.cc:299] Negotiation successful
0810 00:45:13.596480 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":396,"threads_started":1}
I20260810 00:45:13.597694 32441 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4393557b78b4243971110a7fb5f330f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4393557b78b4243971110a7fb5f330f" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 33847 } }
I20260810 00:45:13.598009 32441 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4393557b78b4243971110a7fb5f330f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:45:13.598155 32441 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4393557b78b4243971110a7fb5f330f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:45:13.598368 32441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4393557b78b4243971110a7fb5f330f [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:13.601459 32441 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4393557b78b4243971110a7fb5f330f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4393557b78b4243971110a7fb5f330f" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 33847 } }
I20260810 00:45:13.601943 32441 leader_election.cc:302] T 00000000000000000000000000000000 P a4393557b78b4243971110a7fb5f330f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4393557b78b4243971110a7fb5f330f; no voters:
I20260810 00:45:13.602550 32441 leader_election.cc:288] T 00000000000000000000000000000000 P a4393557b78b4243971110a7fb5f330f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:45:13.602692 32448 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4393557b78b4243971110a7fb5f330f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:45:13.603495 32448 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4393557b78b4243971110a7fb5f330f [term 1 LEADER]: Becoming Leader. State: Replica: a4393557b78b4243971110a7fb5f330f, State: Running, Role: LEADER
I20260810 00:45:13.604031 32448 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4393557b78b4243971110a7fb5f330f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4393557b78b4243971110a7fb5f330f" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 33847 } }
I20260810 00:45:13.604622 32441 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4393557b78b4243971110a7fb5f330f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:45:13.606925 32451 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4393557b78b4243971110a7fb5f330f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4393557b78b4243971110a7fb5f330f. Latest consensus state: current_term: 1 leader_uuid: "a4393557b78b4243971110a7fb5f330f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4393557b78b4243971110a7fb5f330f" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 33847 } } }
I20260810 00:45:13.607333 32451 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4393557b78b4243971110a7fb5f330f [sys.catalog]: This master's current role is: LEADER
I20260810 00:45:13.607087 32450 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4393557b78b4243971110a7fb5f330f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4393557b78b4243971110a7fb5f330f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4393557b78b4243971110a7fb5f330f" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 33847 } } }
I20260810 00:45:13.607465 32450 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4393557b78b4243971110a7fb5f330f [sys.catalog]: This master's current role is: LEADER
I20260810 00:45:13.608337 32458 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:45:13.614754 32458 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:45:13.624449 32458 catalog_manager.cc:1347] Generated new cluster ID: 2e21781b14784763a5d4ea953e2b7a22
I20260810 00:45:13.624553 32458 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:45:13.650161 32458 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:45:13.652001 32458 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:45:13.669253 32458 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4393557b78b4243971110a7fb5f330f: Generated new TSK 0
I20260810 00:45:13.669997 32458 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.6.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab.
Entry for principal kudu/127.28.6.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab.
I20260810 00:45:13.716766 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.1:0
--local_ip_for_outbound_sockets=127.28.6.1
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:33847
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:37131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:45:14.052452 32474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:14.052712 32474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:14.052871 32474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:14.059937 32474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:14.060039 32474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:14.060223 32474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.1
I20260810 00:45:14.069474 32474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:37131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:33847
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.1
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:14.070652 32474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:14.071933 32474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:14.085147 32484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:14.085259 32485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:14.085259 32487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:14.086349 32474 server_base.cc:1034] running on GCE node
I20260810 00:45:14.086969 32474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:14.087846 32474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:14.089047 32474 hybrid_clock.cc:648] HybridClock initialized: now 1786322714089004 us; error 33 us; skew 500 ppm
I20260810 00:45:14.089360 32474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:45:14 dist-test-slave-1nl6 krb5kdc[32328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322714, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:14.092694 32474 init.cc:377] Logged in from keytab as kudu/127.28.6.1@KRBTEST.COM (short username kudu)
I20260810 00:45:14.094177 32474 webserver.cc:460] Webserver started at http://127.28.6.1:34285/ using document root <none> and password file <none>
I20260810 00:45:14.094825 32474 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:14.094992 32474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:14.095311 32474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:14.098210 32474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-0/data/instance:
uuid: "012013d7b37c4d4d944bd2e3e5f0c45e"
format_stamp: "Formatted at 2026-08-10 00:45:14 on dist-test-slave-1nl6"
server_key: "8c94d04d48142e99028b783a7fc5b91a"
server_key_iv: "6e0588d3a401500645e08d083aa7482f"
server_key_version: "encryptionkey@0"
I20260810 00:45:14.099174 32474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance:
uuid: "012013d7b37c4d4d944bd2e3e5f0c45e"
format_stamp: "Formatted at 2026-08-10 00:45:14 on dist-test-slave-1nl6"
server_key: "8c94d04d48142e99028b783a7fc5b91a"
server_key_iv: "6e0588d3a401500645e08d083aa7482f"
server_key_version: "encryptionkey@0"
I20260810 00:45:14.104053 32474 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 00:45:14.107645 32495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:14.109025 32474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 00:45:14.109201 32474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-0/wal
uuid: "012013d7b37c4d4d944bd2e3e5f0c45e"
format_stamp: "Formatted at 2026-08-10 00:45:14 on dist-test-slave-1nl6"
server_key: "8c94d04d48142e99028b783a7fc5b91a"
server_key_iv: "6e0588d3a401500645e08d083aa7482f"
server_key_version: "encryptionkey@0"
I20260810 00:45:14.109396 32474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:14.121070 32474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:14.124747 32474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:14.125036 32474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:14.125943 32474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:45:14.128235 32474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:45:14.128348 32474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:14.128476 32474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:45:14.128551 32474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:14.163586 32474 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.1:34515
I20260810 00:45:14.163652 32659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.1:34515 every 8 connection(s)
I20260810 00:45:14.164849 32474 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
I20260810 00:45:14.167413 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 32474
I20260810 00:45:14.167641 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance
I20260810 00:45:14.168329 28696 external_mini_cluster.cc:1428] Setting key a6befa67623e04b328a1521055ef9330
Aug 10 00:45:14 dist-test-slave-1nl6 krb5kdc[32328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322714, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:45:14.188159 32663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:14.169161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:38815 (local address 127.28.6.62:33847)
0810 00:45:14.169779 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:14.169783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:14.169804 (+ 21us) server_negotiation.cc:407] Connection header received
0810 00:45:14.171369 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:14.171373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:14.171434 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:14.171523 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:14.173651 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:14.174441 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:14.177013 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:14.177420 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:14.177516 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:14.181060 (+ 3544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:14.181071 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:14.181079 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:14.181124 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:14.184725 (+ 3601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:14.185437 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:14.185446 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:14.185453 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:14.185604 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:14.186037 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:14.186045 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:14.186051 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:14.186463 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:14.186642 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:14.187417 (+ 775us) server_negotiation.cc:299] Negotiation successful
0810 00:45:14.187680 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":427,"threads_started":1}
I20260810 00:45:14.188654 32662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:14.167853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:33847 (local address 127.28.6.1:38815)
0810 00:45:14.168687 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:14.168743 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:14.171064 (+ 2321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:14.171812 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:14.171824 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:14.172648 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:14.173472 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:14.173500 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:14.174604 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:14.174613 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:14.176766 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:14.176784 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:14.177651 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:14.177660 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:14.177945 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:14.178626 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:14.178648 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:14.180838 (+ 2190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:14.184865 (+ 4027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:14.184872 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:14.184883 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:14.185277 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:14.185725 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:14.185728 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:14.185730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:14.185900 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:14.186613 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:14.186618 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:14.186972 (+ 354us) client_negotiation.cc:769] Sending connection context
0810 00:45:14.187226 (+ 254us) client_negotiation.cc:240] Negotiation successful
0810 00:45:14.187576 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":348,"threads_started":1}
I20260810 00:45:14.190200 32660 heartbeater.cc:347] Connected to a master server at 127.28.6.62:33847
I20260810 00:45:14.190528 32660 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.28.6.2@KRBTEST.COM; defaulting to no policy
I20260810 00:45:14.191285 32660 heartbeater.cc:511] Master 127.28.6.62:33847 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.2@KRBTEST.COM" created.
I20260810 00:45:14.193508 32386 ts_manager.cc:194] Registered new tserver with Master: 012013d7b37c4d4d944bd2e3e5f0c45e (127.28.6.1:34515)
I20260810 00:45:14.195021 32386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.1@KRBTEST.COM'} at 127.28.6.1:38815
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab.
Entry for principal kudu/127.28.6.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab.
I20260810 00:45:14.216323 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.2:0
--local_ip_for_outbound_sockets=127.28.6.2
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:33847
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:37131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:45:14.540515 32667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:14.540834 32667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:14.541023 32667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:14.548051 32667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:14.548192 32667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:14.548348 32667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.2
I20260810 00:45:14.557507 32667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:37131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:33847
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.2
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:14.558617 32667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:14.559849 32667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:14.573994 32675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:14.574036 32673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:14.575224 32672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:14.577920 32667 server_base.cc:1034] running on GCE node
I20260810 00:45:14.578584 32667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:14.579658 32667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:14.580889 32667 hybrid_clock.cc:648] HybridClock initialized: now 1786322714580855 us; error 33 us; skew 500 ppm
I20260810 00:45:14.581156 32667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:45:14 dist-test-slave-1nl6 krb5kdc[32328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322714, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:14.584245 32667 init.cc:377] Logged in from keytab as kudu/127.28.6.2@KRBTEST.COM (short username kudu)
I20260810 00:45:14.585755 32667 webserver.cc:460] Webserver started at http://127.28.6.2:36605/ using document root <none> and password file <none>
I20260810 00:45:14.586444 32667 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:14.586597 32667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:14.586966 32667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:14.589782 32667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-1/data/instance:
uuid: "dcc24fcd1a8c4e51b8d8c1515f8c1bc9"
format_stamp: "Formatted at 2026-08-10 00:45:14 on dist-test-slave-1nl6"
server_key: "c9e8193f4924f36ecc72c74466564ffa"
server_key_iv: "e02a93a09987349537c1c7288333399b"
server_key_version: "encryptionkey@0"
I20260810 00:45:14.590643 32667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance:
uuid: "dcc24fcd1a8c4e51b8d8c1515f8c1bc9"
format_stamp: "Formatted at 2026-08-10 00:45:14 on dist-test-slave-1nl6"
server_key: "c9e8193f4924f36ecc72c74466564ffa"
server_key_iv: "e02a93a09987349537c1c7288333399b"
server_key_version: "encryptionkey@0"
I20260810 00:45:14.595749 32667 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 00:45:14.599206 32683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:14.600481 32667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 00:45:14.600662 32667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-1/wal
uuid: "dcc24fcd1a8c4e51b8d8c1515f8c1bc9"
format_stamp: "Formatted at 2026-08-10 00:45:14 on dist-test-slave-1nl6"
server_key: "c9e8193f4924f36ecc72c74466564ffa"
server_key_iv: "e02a93a09987349537c1c7288333399b"
server_key_version: "encryptionkey@0"
I20260810 00:45:14.600886 32667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:14.619835 32667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:14.623625 32667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:14.623925 32667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:14.624883 32667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:45:14.626204 32667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:45:14.626312 32667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:14.626478 32667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:45:14.626554 32667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:14.661244 32667 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.2:43601
I20260810 00:45:14.661295 370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.2:43601 every 8 connection(s)
I20260810 00:45:14.662524 32667 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
I20260810 00:45:14.663045 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 32667
I20260810 00:45:14.663269 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance
I20260810 00:45:14.663892 28696 external_mini_cluster.cc:1428] Setting key e3c23315630ed944e658ed6e4c7c65d0
Aug 10 00:45:14 dist-test-slave-1nl6 krb5kdc[32328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322714, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:45:14.685240 32663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:14.665310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:46835 (local address 127.28.6.62:33847)
0810 00:45:14.665482 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:14.665486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:14.666872 (+ 1386us) server_negotiation.cc:407] Connection header received
0810 00:45:14.669216 (+ 2344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:14.669221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:14.669310 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:14.669408 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:14.671055 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:14.671752 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:14.673710 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:14.674121 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:14.674240 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:14.678345 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:14.678350 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:14.678354 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:14.678384 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:14.681793 (+ 3409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:14.682540 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:14.682548 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:14.682555 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:14.682746 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:14.683325 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:14.683333 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:14.683340 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:14.683722 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:14.683898 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:14.684555 (+ 657us) server_negotiation.cc:299] Negotiation successful
0810 00:45:14.684800 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 00:45:14.685801 373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:14.665898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:33847 (local address 127.28.6.2:46835)
0810 00:45:14.666716 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:14.666754 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:14.668962 (+ 2208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:14.669783 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:14.669791 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:14.670302 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:14.670846 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:14.670865 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:14.671920 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:14.671926 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:14.673544 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:14.673562 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:14.674361 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:14.674372 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:14.674693 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:14.675440 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:14.675479 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:14.678104 (+ 2625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:14.681935 (+ 3831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:14.681941 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:14.681952 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:14.682376 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:14.682879 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:14.682882 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:14.682884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:14.683180 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:14.683850 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:14.683855 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:14.684133 (+ 278us) client_negotiation.cc:769] Sending connection context
0810 00:45:14.684395 (+ 262us) client_negotiation.cc:240] Negotiation successful
0810 00:45:14.684752 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":322,"threads_started":1}
WARNING: no policy specified for kudu/127.28.6.3@KRBTEST.COM; defaulting to no policy
I20260810 00:45:14.687326 371 heartbeater.cc:347] Connected to a master server at 127.28.6.62:33847
I20260810 00:45:14.687629 371 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.3@KRBTEST.COM" created.
I20260810 00:45:14.688371 371 heartbeater.cc:511] Master 127.28.6.62:33847 requested a full tablet report, sending...
I20260810 00:45:14.689833 32386 ts_manager.cc:194] Registered new tserver with Master: dcc24fcd1a8c4e51b8d8c1515f8c1bc9 (127.28.6.2:43601)
I20260810 00:45:14.690652 32386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.2@KRBTEST.COM'} at 127.28.6.2:46835
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab.
Entry for principal kudu/127.28.6.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab.
I20260810 00:45:14.711927 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.3:0
--local_ip_for_outbound_sockets=127.28.6.3
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:33847
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:37131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:45:15.044329 377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:15.044651 377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:15.044835 377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:15.051894 377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:15.052047 377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:15.052253 377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.3
I20260810 00:45:15.062438 377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:37131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:33847
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.3
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:15.063932 377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:15.065308 377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:15.078666 390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:15.078830 388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:15.078845 387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:15.079712 377 server_base.cc:1034] running on GCE node
I20260810 00:45:15.080317 377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:15.081254 377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:15.082472 377 hybrid_clock.cc:648] HybridClock initialized: now 1786322715082413 us; error 54 us; skew 500 ppm
I20260810 00:45:15.082779 377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:45:15 dist-test-slave-1nl6 krb5kdc[32328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322715, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:15.085963 377 init.cc:377] Logged in from keytab as kudu/127.28.6.3@KRBTEST.COM (short username kudu)
I20260810 00:45:15.087488 377 webserver.cc:460] Webserver started at http://127.28.6.3:42245/ using document root <none> and password file <none>
I20260810 00:45:15.088173 377 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:15.088325 377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:15.088642 377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:15.091630 377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-2/data/instance:
uuid: "5c27a70ddc8d4849a69c2868240ff723"
format_stamp: "Formatted at 2026-08-10 00:45:15 on dist-test-slave-1nl6"
server_key: "d8fb63f74b5a129c37672b052b92b7c4"
server_key_iv: "b2b87c03c9bf048bc46eaa19adfbbc68"
server_key_version: "encryptionkey@0"
I20260810 00:45:15.092460 377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance:
uuid: "5c27a70ddc8d4849a69c2868240ff723"
format_stamp: "Formatted at 2026-08-10 00:45:15 on dist-test-slave-1nl6"
server_key: "d8fb63f74b5a129c37672b052b92b7c4"
server_key_iv: "b2b87c03c9bf048bc46eaa19adfbbc68"
server_key_version: "encryptionkey@0"
I20260810 00:45:15.097515 377 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 00:45:15.101009 398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:15.102288 377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 00:45:15.102468 377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-2/wal
uuid: "5c27a70ddc8d4849a69c2868240ff723"
format_stamp: "Formatted at 2026-08-10 00:45:15 on dist-test-slave-1nl6"
server_key: "d8fb63f74b5a129c37672b052b92b7c4"
server_key_iv: "b2b87c03c9bf048bc46eaa19adfbbc68"
server_key_version: "encryptionkey@0"
I20260810 00:45:15.102676 377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:15.126721 377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:15.130787 377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:15.131108 377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:15.132082 377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:45:15.133487 377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:45:15.133594 377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:15.133759 377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:45:15.133836 377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:15.168424 377 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.3:46221
I20260810 00:45:15.168486 558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.3:46221 every 8 connection(s)
I20260810 00:45:15.169652 377 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
I20260810 00:45:15.171665 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 377
I20260810 00:45:15.171885 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance
I20260810 00:45:15.172509 28696 external_mini_cluster.cc:1428] Setting key f2d149dd617038b61d4d012f01b89dee
Aug 10 00:45:15 dist-test-slave-1nl6 krb5kdc[32328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322715, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:45:15.189487 32663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:15.172543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:48169 (local address 127.28.6.62:33847)
0810 00:45:15.172746 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:15.172750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:15.174172 (+ 1422us) server_negotiation.cc:407] Connection header received
0810 00:45:15.176080 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:15.176084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:15.176147 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:15.176228 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:15.177613 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:15.178339 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:15.179918 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:15.180350 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:15.180444 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:15.183731 (+ 3287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:15.183736 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:15.183740 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:15.183768 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:15.186532 (+ 2764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:15.187474 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:15.187479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:15.187483 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:15.187648 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:15.188113 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:15.188117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:15.188119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:15.188378 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:15.188489 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:15.189100 (+ 611us) server_negotiation.cc:299] Negotiation successful
0810 00:45:15.189249 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260810 00:45:15.190421 562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:15.173129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:33847 (local address 127.28.6.3:48169)
0810 00:45:15.173978 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:15.174039 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:15.175825 (+ 1786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:15.176440 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:15.176447 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:15.176929 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:15.177470 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:15.177487 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:15.178478 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:15.178481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:15.179768 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:15.179782 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:15.180539 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:15.180544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:15.180735 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:15.181431 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:15.181453 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:15.183542 (+ 2089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:15.186723 (+ 3181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:15.186734 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:15.186750 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:15.187321 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:15.187771 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:15.187778 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:15.187784 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:15.187992 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:15.188507 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:15.188513 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:15.188788 (+ 275us) client_negotiation.cc:769] Sending connection context
0810 00:45:15.188994 (+ 206us) client_negotiation.cc:240] Negotiation successful
0810 00:45:15.189340 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":367,"threads_started":1}
I20260810 00:45:15.191843 559 heartbeater.cc:347] Connected to a master server at 127.28.6.62:33847
I20260810 00:45:15.192101 559 heartbeater.cc:464] Registering TS with master...
I20260810 00:45:15.192731 559 heartbeater.cc:511] Master 127.28.6.62:33847 requested a full tablet report, sending...
I20260810 00:45:15.194093 32386 ts_manager.cc:194] Registered new tserver with Master: 5c27a70ddc8d4849a69c2868240ff723 (127.28.6.3:46221)
I20260810 00:45:15.194818 32386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.3@KRBTEST.COM'} at 127.28.6.3:48169
I20260810 00:45:15.197644 32660 heartbeater.cc:503] Master 127.28.6.62:33847 was elected leader, sending a full tablet report...
I20260810 00:45:15.200191 28696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 00:45:15.235553 32663 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:45:15.233009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39824 (local address 127.28.6.62:33847)
0810 00:45:15.233190 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:15.233195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:15.233762 (+ 567us) server_negotiation.cc:407] Connection header received
0810 00:45:15.233916 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:15.233920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:15.233985 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:15.234081 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:15.235288 (+ 1207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39824: BlockingRecv error: recv got EOF from 127.0.0.1:39824 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 00:45:15.238623 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 32474
W20260810 00:45:15.241374 28696 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 32474: Not authorized: Client connection negotiation failed: client connection to 127.28.6.1:34515: server requires authentication, but client does not have Kerberos credentials available
I20260810 00:45:15.241533 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 32474
I20260810 00:45:15.261804 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 32667
W20260810 00:45:15.264266 28696 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 32667: Not authorized: Client connection negotiation failed: client connection to 127.28.6.2:43601: server requires authentication, but client does not have Kerberos credentials available
I20260810 00:45:15.264431 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 32667
I20260810 00:45:15.288640 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 377
W20260810 00:45:15.291318 28696 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 377: Not authorized: Client connection negotiation failed: client connection to 127.28.6.3:46221: server requires authentication, but client does not have Kerberos credentials available
I20260810 00:45:15.291435 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 377
I20260810 00:45:15.316336 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 32343
I20260810 00:45:15.317556 32430 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:45:15.421765 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 32343
2026-08-10T00:45:15Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3775 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:45:15 dist-test-slave-1nl6 krb5kdc[583](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:45:15 dist-test-slave-1nl6 krb5kdc[583](info): set up 2 sockets
Aug 10 00:45:15 dist-test-slave-1nl6 krb5kdc[583](info): commencing operation
krb5kdc: starting...
W20260810 00:45:16.699012 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.224s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:45:16 dist-test-slave-1nl6 krb5kdc[583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:45:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:45:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.6.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
I20260810 00:45:16.892776 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:45161
--webserver_interface=127.28.6.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:36237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:45161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:45:17.215700 602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:17.215962 602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:17.216151 602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:17.223080 602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:45:17.223208 602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:17.223297 602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:17.223409 602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:45:17.223501 602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:45:17.232820 602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:36237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:45161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:45161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:17.234009 602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:17.235311 602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:17.246590 608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:17.246691 609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:17.246642 611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:17.247828 602 server_base.cc:1034] running on GCE node
I20260810 00:45:17.248561 602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:17.249876 602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:17.251103 602 hybrid_clock.cc:648] HybridClock initialized: now 1786322717251069 us; error 45 us; skew 500 ppm
I20260810 00:45:17.251451 602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:45:17 dist-test-slave-1nl6 krb5kdc[583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322717, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:17.258261 602 init.cc:377] Logged in from keytab as kudu/127.28.6.62@KRBTEST.COM (short username kudu)
I20260810 00:45:17.259783 602 webserver.cc:460] Webserver started at http://127.28.6.62:41863/ using document root <none> and password file <none>
I20260810 00:45:17.260458 602 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:17.260607 602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:17.260924 602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:17.263921 602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "ed57798b37ac45418fe60c5ae19955d8"
format_stamp: "Formatted at 2026-08-10 00:45:17 on dist-test-slave-1nl6"
server_key: "6a600dfbf613f38600f1242469b588e9"
server_key_iv: "00137ed0a4d55bbd2876006e510db3d0"
server_key_version: "encryptionkey@0"
I20260810 00:45:17.264700 602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "ed57798b37ac45418fe60c5ae19955d8"
format_stamp: "Formatted at 2026-08-10 00:45:17 on dist-test-slave-1nl6"
server_key: "6a600dfbf613f38600f1242469b588e9"
server_key_iv: "00137ed0a4d55bbd2876006e510db3d0"
server_key_version: "encryptionkey@0"
I20260810 00:45:17.269632 602 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 00:45:17.273005 621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:17.274221 602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 00:45:17.274407 602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "ed57798b37ac45418fe60c5ae19955d8"
format_stamp: "Formatted at 2026-08-10 00:45:17 on dist-test-slave-1nl6"
server_key: "6a600dfbf613f38600f1242469b588e9"
server_key_iv: "00137ed0a4d55bbd2876006e510db3d0"
server_key_version: "encryptionkey@0"
I20260810 00:45:17.274601 602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:17.293322 602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:17.297108 602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:17.297399 602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:17.315804 602 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:45161
I20260810 00:45:17.315853 702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:45161 every 8 connection(s)
I20260810 00:45:17.317114 602 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:45:17.319046 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 602
I20260810 00:45:17.319252 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:45:17.320147 28696 external_mini_cluster.cc:1428] Setting key 404a27d1dc39d9ac2adb0e0e439fa2c3
I20260810 00:45:17.321599 703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 00:45:17 dist-test-slave-1nl6 krb5kdc[583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:45:17.335795 703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed57798b37ac45418fe60c5ae19955d8: Bootstrap starting.
I20260810 00:45:17.340111 703 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed57798b37ac45418fe60c5ae19955d8: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:17.341419 703 log.cc:824] T 00000000000000000000000000000000 P ed57798b37ac45418fe60c5ae19955d8: Log is configured to *not* fsync() on all Append() calls
I20260810 00:45:17.344743 706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:17.321985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33088 (local address 127.28.6.62:45161)
0810 00:45:17.322771 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:17.322781 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:17.322819 (+ 38us) server_negotiation.cc:407] Connection header received
0810 00:45:17.324425 (+ 1606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:17.324445 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:17.324818 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:17.325169 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:17.327061 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:17.328451 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:17.330276 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:17.330979 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:17.331125 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:17.335890 (+ 4765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:17.335902 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:17.335921 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:17.335964 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:17.339850 (+ 3886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:17.340598 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:17.340607 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:17.340615 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:17.340827 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:17.341187 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:17.341193 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:17.341202 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:17.341919 (+ 717us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:17.342285 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:17.342843 (+ 558us) server_negotiation.cc:299] Negotiation successful
0810 00:45:17.343367 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":363,"threads_started":1}
I20260810 00:45:17.345124 703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed57798b37ac45418fe60c5ae19955d8: No bootstrap required, opened a new log
I20260810 00:45:17.348670 28696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 00:45:17.353400 703 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed57798b37ac45418fe60c5ae19955d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed57798b37ac45418fe60c5ae19955d8" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 45161 } }
I20260810 00:45:17.353754 703 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed57798b37ac45418fe60c5ae19955d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:17.353873 703 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed57798b37ac45418fe60c5ae19955d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed57798b37ac45418fe60c5ae19955d8, State: Initialized, Role: FOLLOWER
I20260810 00:45:17.354460 703 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed57798b37ac45418fe60c5ae19955d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed57798b37ac45418fe60c5ae19955d8" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 45161 } }
I20260810 00:45:17.354712 703 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed57798b37ac45418fe60c5ae19955d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:45:17.354848 703 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed57798b37ac45418fe60c5ae19955d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:45:17.355122 703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed57798b37ac45418fe60c5ae19955d8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:17.358043 703 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed57798b37ac45418fe60c5ae19955d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed57798b37ac45418fe60c5ae19955d8" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 45161 } }
I20260810 00:45:17.358522 703 leader_election.cc:302] T 00000000000000000000000000000000 P ed57798b37ac45418fe60c5ae19955d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed57798b37ac45418fe60c5ae19955d8; no voters:
I20260810 00:45:17.359238 703 leader_election.cc:288] T 00000000000000000000000000000000 P ed57798b37ac45418fe60c5ae19955d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:45:17.359349 710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed57798b37ac45418fe60c5ae19955d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:45:17.360592 710 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed57798b37ac45418fe60c5ae19955d8 [term 1 LEADER]: Becoming Leader. State: Replica: ed57798b37ac45418fe60c5ae19955d8, State: Running, Role: LEADER
I20260810 00:45:17.361208 710 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed57798b37ac45418fe60c5ae19955d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed57798b37ac45418fe60c5ae19955d8" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 45161 } }
I20260810 00:45:17.361671 703 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed57798b37ac45418fe60c5ae19955d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:45:17.364158 713 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed57798b37ac45418fe60c5ae19955d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed57798b37ac45418fe60c5ae19955d8. Latest consensus state: current_term: 1 leader_uuid: "ed57798b37ac45418fe60c5ae19955d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed57798b37ac45418fe60c5ae19955d8" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 45161 } } }
I20260810 00:45:17.364629 713 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed57798b37ac45418fe60c5ae19955d8 [sys.catalog]: This master's current role is: LEADER
I20260810 00:45:17.365353 712 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed57798b37ac45418fe60c5ae19955d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed57798b37ac45418fe60c5ae19955d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed57798b37ac45418fe60c5ae19955d8" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 45161 } } }
I20260810 00:45:17.365684 712 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed57798b37ac45418fe60c5ae19955d8 [sys.catalog]: This master's current role is: LEADER
I20260810 00:45:17.372767 719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:45:17.379701 719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:45:17.392328 719 catalog_manager.cc:1347] Generated new cluster ID: 0765ce9370024d28afc90a756263535c
I20260810 00:45:17.392434 719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:45:17.404608 706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:17.395520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57626 (local address 127.28.6.62:45161)
0810 00:45:17.395712 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:17.395721 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:17.396236 (+ 515us) server_negotiation.cc:407] Connection header received
0810 00:45:17.396439 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:17.396446 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:17.396531 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:17.396646 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:17.397979 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:17.399166 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:17.400922 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:17.401588 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:17.401726 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:17.402873 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:17.402880 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:17.402886 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:45:17.403750 (+ 864us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:17.403810 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:45:17.403855 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:17.403965 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:17.404117 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 00:45:17.404210 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 00:45:17.416754 719 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:45:17.418210 719 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:45:17.427995 719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed57798b37ac45418fe60c5ae19955d8: Generated new TSK 0
I20260810 00:45:17.428736 719 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 00:45:17.437099 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 602
I20260810 00:45:17.438587 698 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:45:17.527249 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 602
2026-08-10T00:45:17Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2098 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:45:17 dist-test-slave-1nl6 krb5kdc[752](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:45:17 dist-test-slave-1nl6 krb5kdc[752](info): set up 2 sockets
Aug 10 00:45:17 dist-test-slave-1nl6 krb5kdc[752](info): commencing operation
krb5kdc: starting...
W20260810 00:45:18.816433 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.244s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:45:18 dist-test-slave-1nl6 krb5kdc[752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:45:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:45:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.6.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
I20260810 00:45:19.008351 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:38669
--webserver_interface=127.28.6.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:38669
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:45:19.328393 773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:19.328717 773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:19.328902 773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:19.335872 773 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:45:19.336011 773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:19.336122 773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:19.336230 773 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:45:19.336324 773 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:45:19.345696 773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:38669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:38669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:19.347020 773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:19.348284 773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:19.359797 780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:19.359866 784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:19.359932 781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:19.360757 773 server_base.cc:1034] running on GCE node
I20260810 00:45:19.361483 773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:19.362800 773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:19.364037 773 hybrid_clock.cc:648] HybridClock initialized: now 1786322719363993 us; error 67 us; skew 500 ppm
I20260810 00:45:19.364389 773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:45:19 dist-test-slave-1nl6 krb5kdc[752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322719, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:19.367636 773 init.cc:377] Logged in from keytab as kudu/127.28.6.62@KRBTEST.COM (short username kudu)
I20260810 00:45:19.369076 773 webserver.cc:460] Webserver started at http://127.28.6.62:43863/ using document root <none> and password file <none>
I20260810 00:45:19.369745 773 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:19.369896 773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:19.370208 773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:19.373338 773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "9ca6de96617d49b5a3cf56a551b5b0d4"
format_stamp: "Formatted at 2026-08-10 00:45:19 on dist-test-slave-1nl6"
server_key: "be09509b0a4714768ef38d2ff7f1c57d"
server_key_iv: "0937e977a2859e0482d963f1d9365f86"
server_key_version: "encryptionkey@0"
I20260810 00:45:19.374090 773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "9ca6de96617d49b5a3cf56a551b5b0d4"
format_stamp: "Formatted at 2026-08-10 00:45:19 on dist-test-slave-1nl6"
server_key: "be09509b0a4714768ef38d2ff7f1c57d"
server_key_iv: "0937e977a2859e0482d963f1d9365f86"
server_key_version: "encryptionkey@0"
I20260810 00:45:19.379252 773 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260810 00:45:19.382511 795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:19.383841 773 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 00:45:19.384029 773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "9ca6de96617d49b5a3cf56a551b5b0d4"
format_stamp: "Formatted at 2026-08-10 00:45:19 on dist-test-slave-1nl6"
server_key: "be09509b0a4714768ef38d2ff7f1c57d"
server_key_iv: "0937e977a2859e0482d963f1d9365f86"
server_key_version: "encryptionkey@0"
I20260810 00:45:19.384222 773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:19.402292 773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:19.406522 773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:19.406790 773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:19.426347 773 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:38669
I20260810 00:45:19.426378 871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:38669 every 8 connection(s)
I20260810 00:45:19.427693 773 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:45:19.431265 872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:19.436403 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 773
I20260810 00:45:19.436626 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:45:19.437398 28696 external_mini_cluster.cc:1428] Setting key 94237ab1206d3e5ca4d9a705dddbef57
I20260810 00:45:19.442476 872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ca6de96617d49b5a3cf56a551b5b0d4: Bootstrap starting.
I20260810 00:45:19.445410 872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ca6de96617d49b5a3cf56a551b5b0d4: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:19.446369 872 log.cc:824] T 00000000000000000000000000000000 P 9ca6de96617d49b5a3cf56a551b5b0d4: Log is configured to *not* fsync() on all Append() calls
I20260810 00:45:19.448993 872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ca6de96617d49b5a3cf56a551b5b0d4: No bootstrap required, opened a new log
Aug 10 00:45:19 dist-test-slave-1nl6 krb5kdc[752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:45:19.457564 872 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ca6de96617d49b5a3cf56a551b5b0d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ca6de96617d49b5a3cf56a551b5b0d4" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 38669 } }
I20260810 00:45:19.458017 872 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ca6de96617d49b5a3cf56a551b5b0d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:19.458145 872 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ca6de96617d49b5a3cf56a551b5b0d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ca6de96617d49b5a3cf56a551b5b0d4, State: Initialized, Role: FOLLOWER
I20260810 00:45:19.458698 872 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ca6de96617d49b5a3cf56a551b5b0d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ca6de96617d49b5a3cf56a551b5b0d4" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 38669 } }
I20260810 00:45:19.458949 872 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ca6de96617d49b5a3cf56a551b5b0d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:45:19.459105 872 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ca6de96617d49b5a3cf56a551b5b0d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:45:19.459288 872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ca6de96617d49b5a3cf56a551b5b0d4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:19.459534 875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:19.439459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45240 (local address 127.28.6.62:38669)
0810 00:45:19.440326 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:19.440338 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:19.440388 (+ 50us) server_negotiation.cc:407] Connection header received
0810 00:45:19.442005 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:19.442027 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:19.442385 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:19.442788 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:19.444220 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:19.445442 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:19.446770 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:19.447524 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:19.447714 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:19.452023 (+ 4309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:19.452031 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:19.452044 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:19.452084 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:19.455210 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:19.455999 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:19.456005 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:19.456011 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:19.456176 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:19.456587 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:19.456590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:19.456592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:19.457061 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:19.457289 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:19.457720 (+ 431us) server_negotiation.cc:299] Negotiation successful
0810 00:45:19.458268 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":368,"threads_started":1}
I20260810 00:45:19.461591 872 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ca6de96617d49b5a3cf56a551b5b0d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ca6de96617d49b5a3cf56a551b5b0d4" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 38669 } }
I20260810 00:45:19.462144 872 leader_election.cc:302] T 00000000000000000000000000000000 P 9ca6de96617d49b5a3cf56a551b5b0d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ca6de96617d49b5a3cf56a551b5b0d4; no voters:
I20260810 00:45:19.462738 872 leader_election.cc:288] T 00000000000000000000000000000000 P 9ca6de96617d49b5a3cf56a551b5b0d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:45:19.462865 877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ca6de96617d49b5a3cf56a551b5b0d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:45:19.464124 877 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ca6de96617d49b5a3cf56a551b5b0d4 [term 1 LEADER]: Becoming Leader. State: Replica: 9ca6de96617d49b5a3cf56a551b5b0d4, State: Running, Role: LEADER
I20260810 00:45:19.464685 877 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ca6de96617d49b5a3cf56a551b5b0d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ca6de96617d49b5a3cf56a551b5b0d4" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 38669 } }
I20260810 00:45:19.464811 872 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ca6de96617d49b5a3cf56a551b5b0d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:45:19.467651 878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ca6de96617d49b5a3cf56a551b5b0d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ca6de96617d49b5a3cf56a551b5b0d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ca6de96617d49b5a3cf56a551b5b0d4" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 38669 } } }
I20260810 00:45:19.468246 878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ca6de96617d49b5a3cf56a551b5b0d4 [sys.catalog]: This master's current role is: LEADER
I20260810 00:45:19.467625 879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ca6de96617d49b5a3cf56a551b5b0d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ca6de96617d49b5a3cf56a551b5b0d4. Latest consensus state: current_term: 1 leader_uuid: "9ca6de96617d49b5a3cf56a551b5b0d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ca6de96617d49b5a3cf56a551b5b0d4" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 38669 } } }
I20260810 00:45:19.468834 879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ca6de96617d49b5a3cf56a551b5b0d4 [sys.catalog]: This master's current role is: LEADER
I20260810 00:45:19.471860 888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:45:19.478349 888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:45:19.486842 888 catalog_manager.cc:1347] Generated new cluster ID: cafa2f0b4864415281dbcd1cb8c627db
I20260810 00:45:19.486979 888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:45:19.506243 888 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:45:19.508111 888 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:45:19.520442 888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ca6de96617d49b5a3cf56a551b5b0d4: Generated new TSK 0
I20260810 00:45:19.521385 888 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.6.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab.
Entry for principal kudu/127.28.6.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab.
I20260810 00:45:19.570377 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.1:0
--local_ip_for_outbound_sockets=127.28.6.1
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:38669
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:45:19.912021 905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:19.912304 905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:19.912503 905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:19.919608 905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:19.919700 905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:19.919900 905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.1
I20260810 00:45:19.928998 905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:38669
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.1
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:19.930219 905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:19.931545 905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:19.944084 914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:19.944111 917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:19.944154 923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:19.945075 905 server_base.cc:1034] running on GCE node
I20260810 00:45:19.945616 905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:19.946514 905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:19.947749 905 hybrid_clock.cc:648] HybridClock initialized: now 1786322719947693 us; error 49 us; skew 500 ppm
I20260810 00:45:19.948040 905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:45:19 dist-test-slave-1nl6 krb5kdc[752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322719, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:19.951176 905 init.cc:377] Logged in from keytab as kudu/127.28.6.1@KRBTEST.COM (short username kudu)
I20260810 00:45:19.952517 905 webserver.cc:460] Webserver started at http://127.28.6.1:34827/ using document root <none> and password file <none>
I20260810 00:45:19.953161 905 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:19.953305 905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:19.953605 905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:19.956585 905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-0/data/instance:
uuid: "88dc983e852d461eb31254364d96ebdd"
format_stamp: "Formatted at 2026-08-10 00:45:19 on dist-test-slave-1nl6"
server_key: "a635cbca3c7ad9c07813a9b7eae48bd6"
server_key_iv: "94dd6c263015923b7068c4c5dcd30223"
server_key_version: "encryptionkey@0"
I20260810 00:45:19.957360 905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance:
uuid: "88dc983e852d461eb31254364d96ebdd"
format_stamp: "Formatted at 2026-08-10 00:45:19 on dist-test-slave-1nl6"
server_key: "a635cbca3c7ad9c07813a9b7eae48bd6"
server_key_iv: "94dd6c263015923b7068c4c5dcd30223"
server_key_version: "encryptionkey@0"
I20260810 00:45:19.962311 905 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 00:45:19.965719 932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:19.966987 905 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 00:45:19.967151 905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-0/wal
uuid: "88dc983e852d461eb31254364d96ebdd"
format_stamp: "Formatted at 2026-08-10 00:45:19 on dist-test-slave-1nl6"
server_key: "a635cbca3c7ad9c07813a9b7eae48bd6"
server_key_iv: "94dd6c263015923b7068c4c5dcd30223"
server_key_version: "encryptionkey@0"
I20260810 00:45:19.967338 905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:20.007529 905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:20.011741 905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:20.012061 905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:20.013078 905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:45:20.014493 905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:45:20.014601 905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:20.014761 905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:45:20.014835 905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:20.048727 905 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.1:37517
I20260810 00:45:20.048816 1090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.1:37517 every 8 connection(s)
I20260810 00:45:20.050051 905 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
I20260810 00:45:20.051702 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 905
I20260810 00:45:20.051919 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance
I20260810 00:45:20.052594 28696 external_mini_cluster.cc:1428] Setting key 8c1fe1e01650f3ea5239839dc0cea1fc
Aug 10 00:45:20 dist-test-slave-1nl6 krb5kdc[752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322719, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:45:20.073979 1094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:20.052803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:60263 (local address 127.28.6.62:38669)
0810 00:45:20.053457 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:20.053465 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:20.054378 (+ 913us) server_negotiation.cc:407] Connection header received
0810 00:45:20.057339 (+ 2961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:20.057343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:20.057403 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:20.057487 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:20.059655 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:20.060361 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:20.062862 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:20.063587 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:20.063735 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:20.067737 (+ 4002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:20.067743 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:20.067746 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:20.067778 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:20.070445 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:20.071522 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:20.071526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:20.071528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:20.071681 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:20.072215 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:20.072218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:20.072220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:20.072466 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:20.072574 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:20.073550 (+ 976us) server_negotiation.cc:299] Negotiation successful
0810 00:45:20.073708 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":257,"threads_started":1}
I20260810 00:45:20.075549 1095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:20.053473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:38669 (local address 127.28.6.1:60263)
0810 00:45:20.054210 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:20.054253 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:20.057006 (+ 2753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:20.057754 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:20.057767 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:20.058603 (+ 836us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:20.059466 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:20.059495 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:20.060522 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:20.060530 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:20.062706 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:20.062719 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:20.063803 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:20.063808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:20.063995 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:20.064687 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:20.064710 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:20.067538 (+ 2828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:20.070642 (+ 3104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:20.070662 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:20.070681 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:20.071396 (+ 715us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:20.071813 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:20.071820 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:20.071826 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:20.072078 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:20.072617 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:20.072628 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:20.073109 (+ 481us) client_negotiation.cc:769] Sending connection context
0810 00:45:20.073455 (+ 346us) client_negotiation.cc:240] Negotiation successful
0810 00:45:20.074036 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":307,"threads_started":1}
WARNING: no policy specified for kudu/127.28.6.2@KRBTEST.COM; defaulting to no policy
I20260810 00:45:20.077277 1091 heartbeater.cc:347] Connected to a master server at 127.28.6.62:38669
I20260810 00:45:20.077651 1091 heartbeater.cc:464] Registering TS with master...
I20260810 00:45:20.078394 1091 heartbeater.cc:511] Master 127.28.6.62:38669 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.2@KRBTEST.COM" created.
I20260810 00:45:20.080651 820 ts_manager.cc:194] Registered new tserver with Master: 88dc983e852d461eb31254364d96ebdd (127.28.6.1:37517)
I20260810 00:45:20.082080 820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.1@KRBTEST.COM'} at 127.28.6.1:60263
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab.
Entry for principal kudu/127.28.6.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab.
I20260810 00:45:20.101709 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.2:0
--local_ip_for_outbound_sockets=127.28.6.2
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:38669
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:45:20.446201 1099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:20.446483 1099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:20.446601 1099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:20.454262 1099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:20.454368 1099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:20.454490 1099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.2
I20260810 00:45:20.464325 1099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:38669
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.2
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:20.465750 1099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:20.467206 1099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:20.481223 1109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:20.481253 1107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:20.481307 1106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:20.482856 1099 server_base.cc:1034] running on GCE node
I20260810 00:45:20.483518 1099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:20.484442 1099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:20.485644 1099 hybrid_clock.cc:648] HybridClock initialized: now 1786322720485603 us; error 37 us; skew 500 ppm
I20260810 00:45:20.485982 1099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:45:20 dist-test-slave-1nl6 krb5kdc[752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322720, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:20.489296 1099 init.cc:377] Logged in from keytab as kudu/127.28.6.2@KRBTEST.COM (short username kudu)
I20260810 00:45:20.490774 1099 webserver.cc:460] Webserver started at http://127.28.6.2:42085/ using document root <none> and password file <none>
I20260810 00:45:20.491508 1099 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:20.491645 1099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:20.491969 1099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:20.494946 1099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-1/data/instance:
uuid: "5533e9138e5b437294b553617aecae18"
format_stamp: "Formatted at 2026-08-10 00:45:20 on dist-test-slave-1nl6"
server_key: "3c3939468d9e08f0f5ed643520c56bd9"
server_key_iv: "49ba8ac00cced199a30d1e8a445522bb"
server_key_version: "encryptionkey@0"
I20260810 00:45:20.495848 1099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance:
uuid: "5533e9138e5b437294b553617aecae18"
format_stamp: "Formatted at 2026-08-10 00:45:20 on dist-test-slave-1nl6"
server_key: "3c3939468d9e08f0f5ed643520c56bd9"
server_key_iv: "49ba8ac00cced199a30d1e8a445522bb"
server_key_version: "encryptionkey@0"
I20260810 00:45:20.501097 1099 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260810 00:45:20.504565 1118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:20.505815 1099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 00:45:20.505980 1099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-1/wal
uuid: "5533e9138e5b437294b553617aecae18"
format_stamp: "Formatted at 2026-08-10 00:45:20 on dist-test-slave-1nl6"
server_key: "3c3939468d9e08f0f5ed643520c56bd9"
server_key_iv: "49ba8ac00cced199a30d1e8a445522bb"
server_key_version: "encryptionkey@0"
I20260810 00:45:20.506156 1099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:20.531209 1099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:20.535135 1099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:20.535418 1099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:20.536377 1099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:45:20.537732 1099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:45:20.537817 1099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:20.537962 1099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:45:20.538035 1099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:20.572218 1099 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.2:45913
I20260810 00:45:20.572252 1278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.2:45913 every 8 connection(s)
I20260810 00:45:20.573455 1099 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
I20260810 00:45:20.581121 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 1099
I20260810 00:45:20.581301 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance
I20260810 00:45:20.581874 28696 external_mini_cluster.cc:1428] Setting key 1613136ca7b422dadfc74e1f0aef41f3
Aug 10 00:45:20 dist-test-slave-1nl6 krb5kdc[752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322720, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:45:20.593957 1282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:20.576071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:50091 (local address 127.28.6.62:38669)
0810 00:45:20.576601 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:20.576608 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:20.577472 (+ 864us) server_negotiation.cc:407] Connection header received
0810 00:45:20.579278 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:20.579283 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:20.579356 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:20.579453 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:20.580915 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:20.582025 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:20.583634 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:20.584080 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:20.584225 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:20.587791 (+ 3566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:20.587796 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:20.587799 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:20.587831 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:20.590580 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:20.591394 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:20.591402 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:20.591407 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:20.591577 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:20.592012 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:20.592020 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:20.592026 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:20.592409 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:20.592567 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:20.593183 (+ 616us) server_negotiation.cc:299] Negotiation successful
0810 00:45:20.593456 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":293,"threads_started":1}
I20260810 00:45:20.594475 1283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:20.576540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:38669 (local address 127.28.6.2:50091)
0810 00:45:20.577306 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:20.577351 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:20.579038 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:20.579663 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:20.579671 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:20.580169 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:20.580740 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:20.580758 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:20.582169 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:20.582173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:20.583482 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:20.583494 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:20.584326 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:20.584334 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:20.584564 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:20.585261 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:20.585291 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:20.587552 (+ 2261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:20.590753 (+ 3201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:20.590760 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:20.590771 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:20.591258 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:20.591690 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:20.591693 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:20.591696 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:20.591884 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:20.592532 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:20.592537 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:20.592811 (+ 274us) client_negotiation.cc:769] Sending connection context
0810 00:45:20.593037 (+ 226us) client_negotiation.cc:240] Negotiation successful
0810 00:45:20.593380 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":285,"threads_started":1}
I20260810 00:45:20.596127 1279 heartbeater.cc:347] Connected to a master server at 127.28.6.62:38669
I20260810 00:45:20.596472 1279 heartbeater.cc:464] Registering TS with master...
I20260810 00:45:20.597390 1279 heartbeater.cc:511] Master 127.28.6.62:38669 requested a full tablet report, sending...
I20260810 00:45:20.599032 820 ts_manager.cc:194] Registered new tserver with Master: 5533e9138e5b437294b553617aecae18 (127.28.6.2:45913)
I20260810 00:45:20.599844 820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.2@KRBTEST.COM'} at 127.28.6.2:50091
WARNING: no policy specified for kudu/127.28.6.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab.
Entry for principal kudu/127.28.6.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab.
I20260810 00:45:20.631891 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.3:0
--local_ip_for_outbound_sockets=127.28.6.3
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:38669
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:45:20.959206 1288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:20.959538 1288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:20.959698 1288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:20.966770 1288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:20.966904 1288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:20.967129 1288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.3
I20260810 00:45:20.976966 1288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:38669
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.3
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:20.978199 1288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:20.979552 1288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:20.992592 1298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:20.992750 1294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:20.992703 1296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:20.993646 1288 server_base.cc:1034] running on GCE node
I20260810 00:45:20.994249 1288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:20.995185 1288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:20.996392 1288 hybrid_clock.cc:648] HybridClock initialized: now 1786322720996353 us; error 31 us; skew 500 ppm
I20260810 00:45:20.996706 1288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:45:20 dist-test-slave-1nl6 krb5kdc[752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322720, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:20.999850 1288 init.cc:377] Logged in from keytab as kudu/127.28.6.3@KRBTEST.COM (short username kudu)
I20260810 00:45:21.002111 1288 webserver.cc:460] Webserver started at http://127.28.6.3:46159/ using document root <none> and password file <none>
I20260810 00:45:21.002791 1288 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:21.002955 1288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:21.003283 1288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:21.006179 1288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-2/data/instance:
uuid: "386523ed73dc48178a3c1a3798838a63"
format_stamp: "Formatted at 2026-08-10 00:45:21 on dist-test-slave-1nl6"
server_key: "55ce0e861bca1a4634578d76ac778541"
server_key_iv: "672ec8067fd428a1e32183a994f529e0"
server_key_version: "encryptionkey@0"
I20260810 00:45:21.007126 1288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance:
uuid: "386523ed73dc48178a3c1a3798838a63"
format_stamp: "Formatted at 2026-08-10 00:45:21 on dist-test-slave-1nl6"
server_key: "55ce0e861bca1a4634578d76ac778541"
server_key_iv: "672ec8067fd428a1e32183a994f529e0"
server_key_version: "encryptionkey@0"
I20260810 00:45:21.012449 1288 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 00:45:21.015903 1310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:21.017145 1288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 00:45:21.017330 1288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-2/wal
uuid: "386523ed73dc48178a3c1a3798838a63"
format_stamp: "Formatted at 2026-08-10 00:45:21 on dist-test-slave-1nl6"
server_key: "55ce0e861bca1a4634578d76ac778541"
server_key_iv: "672ec8067fd428a1e32183a994f529e0"
server_key_version: "encryptionkey@0"
I20260810 00:45:21.017526 1288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:21.068382 1288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:21.073163 1288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:21.073485 1288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:21.074510 1288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:45:21.076119 1288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:45:21.076226 1288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:21.076388 1288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:45:21.076467 1288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:21.084848 1091 heartbeater.cc:503] Master 127.28.6.62:38669 was elected leader, sending a full tablet report...
I20260810 00:45:21.112267 1288 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.3:41669
I20260810 00:45:21.112428 1476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.3:41669 every 8 connection(s)
I20260810 00:45:21.113612 1288 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
I20260810 00:45:21.122051 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 1288
I20260810 00:45:21.122296 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance
I20260810 00:45:21.122884 28696 external_mini_cluster.cc:1428] Setting key 7fe424ac31e0306c1e7da75c865daf6b
Aug 10 00:45:21 dist-test-slave-1nl6 krb5kdc[752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322720, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:45:21.138707 1483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:21.117063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:33975 (local address 127.28.6.62:38669)
0810 00:45:21.117806 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:21.117811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:21.118974 (+ 1163us) server_negotiation.cc:407] Connection header received
0810 00:45:21.121699 (+ 2725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:21.121705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:21.121834 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:21.121961 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:21.124090 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:21.125188 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:21.127098 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:21.127609 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:21.127762 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:21.132346 (+ 4584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:21.132352 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:21.132355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:21.132388 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:21.135021 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:21.135864 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:21.135869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:21.135873 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:21.136000 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:21.136648 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:21.136653 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:21.136657 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:21.136964 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:21.137146 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:21.138043 (+ 897us) server_negotiation.cc:299] Negotiation successful
0810 00:45:21.138310 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":479,"threads_started":1}
I20260810 00:45:21.139770 1484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:21.117569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:38669 (local address 127.28.6.3:33975)
0810 00:45:21.118784 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:21.118832 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:21.121405 (+ 2573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:21.122269 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:21.122281 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:21.122976 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:21.123868 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:21.123889 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:21.125364 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:21.125369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:21.126891 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:21.126954 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:21.127883 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:21.127894 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:21.128163 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:21.128909 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:21.128932 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:21.132056 (+ 3124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:21.135159 (+ 3103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:21.135168 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:21.135185 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:21.135720 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:21.136117 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:21.136122 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:21.136126 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:21.136415 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:21.137114 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:21.137124 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:21.137577 (+ 453us) client_negotiation.cc:769] Sending connection context
0810 00:45:21.137900 (+ 323us) client_negotiation.cc:240] Negotiation successful
0810 00:45:21.138541 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":938,"thread_start_us":622,"threads_started":1}
I20260810 00:45:21.141155 1478 heartbeater.cc:347] Connected to a master server at 127.28.6.62:38669
I20260810 00:45:21.141448 1478 heartbeater.cc:464] Registering TS with master...
I20260810 00:45:21.142122 1478 heartbeater.cc:511] Master 127.28.6.62:38669 requested a full tablet report, sending...
I20260810 00:45:21.143610 820 ts_manager.cc:194] Registered new tserver with Master: 386523ed73dc48178a3c1a3798838a63 (127.28.6.3:41669)
I20260810 00:45:21.144503 820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.3@KRBTEST.COM'} at 127.28.6.3:33975
I20260810 00:45:21.150938 28696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 00:45:21 dist-test-slave-1nl6 krb5kdc[752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322721, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 10 00:45:21 dist-test-slave-1nl6 krb5kdc[752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322721, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:45:21.203106 1483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:21.188509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45250 (local address 127.28.6.62:38669)
0810 00:45:21.188777 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:21.188785 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:21.189283 (+ 498us) server_negotiation.cc:407] Connection header received
0810 00:45:21.189522 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:21.189526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:21.189597 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:21.189727 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:21.191221 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:21.191990 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:21.193315 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:21.193769 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:21.193914 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:21.197836 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:21.197842 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:21.197845 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:21.197880 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:21.200375 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:21.201020 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:21.201026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:21.201030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:21.201159 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:21.201551 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:21.201556 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:21.201560 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:21.201887 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:21.202164 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:21.202511 (+ 347us) server_negotiation.cc:299] Negotiation successful
0810 00:45:21.202769 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260810 00:45:21.203653 820 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45250
I20260810 00:45:21.205289 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 905
Aug 10 00:45:21 dist-test-slave-1nl6 krb5kdc[752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322721, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.6.1@KRBTEST.COM
I20260810 00:45:21.221084 1495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:21.206357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46918 (local address 127.28.6.1:37517)
0810 00:45:21.207012 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:21.207019 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:21.207044 (+ 25us) server_negotiation.cc:407] Connection header received
0810 00:45:21.207115 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:21.207121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:21.207344 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:21.207471 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:21.208686 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:21.209698 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:21.211066 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:21.211616 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:21.211712 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:21.215475 (+ 3763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:21.215485 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:21.215500 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:21.215535 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:21.218446 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:21.219097 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:21.219104 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:21.219106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:21.219232 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:21.219630 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:21.219634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:21.219635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:21.219953 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:21.220180 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:21.220522 (+ 342us) server_negotiation.cc:299] Negotiation successful
0810 00:45:21.220719 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":322,"threads_started":1}
W20260810 00:45:21.222398 1083 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46918
W20260810 00:45:21.223587 28696 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 905: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 00:45:21.223806 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 905
I20260810 00:45:21.247648 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 1099
Aug 10 00:45:21 dist-test-slave-1nl6 krb5kdc[752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322721, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.6.2@KRBTEST.COM
I20260810 00:45:21.263198 1496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:21.248489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50890 (local address 127.28.6.2:45913)
0810 00:45:21.249046 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:21.249053 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:21.249079 (+ 26us) server_negotiation.cc:407] Connection header received
0810 00:45:21.249141 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:21.249147 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:21.249361 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:21.249538 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:21.250770 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:21.251862 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:21.253148 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:21.253702 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:21.253853 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:21.258002 (+ 4149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:21.258013 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:21.258027 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:21.258059 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:21.260712 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:21.261319 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:21.261326 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:21.261329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:21.261458 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:21.261774 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:21.261778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:21.261780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:21.262120 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:21.262378 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:21.262604 (+ 226us) server_negotiation.cc:299] Negotiation successful
0810 00:45:21.262809 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":270,"threads_started":1}
W20260810 00:45:21.264495 1273 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50890
W20260810 00:45:21.265633 28696 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 1099: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 00:45:21.265811 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 1099
I20260810 00:45:21.291105 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 1288
Aug 10 00:45:21 dist-test-slave-1nl6 krb5kdc[752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322721, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.6.3@KRBTEST.COM
I20260810 00:45:21.307165 1498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:21.292221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41436 (local address 127.28.6.3:41669)
0810 00:45:21.293126 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:21.293133 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:21.293162 (+ 29us) server_negotiation.cc:407] Connection header received
0810 00:45:21.293227 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:21.293234 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:21.293441 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:21.293569 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:21.294808 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:21.295926 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:21.297140 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:21.297682 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:21.297791 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:21.301933 (+ 4142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:21.301945 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:21.301959 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:21.302004 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:21.304677 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:21.305279 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:21.305286 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:21.305288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:21.305412 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:21.305738 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:21.305741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:21.305743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:21.306082 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:21.306360 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:21.306556 (+ 196us) server_negotiation.cc:299] Negotiation successful
0810 00:45:21.306791 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":499,"threads_started":1}
W20260810 00:45:21.308543 1467 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41436
W20260810 00:45:21.309583 28696 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 1288: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 00:45:21.309746 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 1288
I20260810 00:45:21.340044 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 773
I20260810 00:45:21.341305 866 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:45:21.444885 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 773
2026-08-10T00:45:21Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3922 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:45:21 dist-test-slave-1nl6 krb5kdc[1507](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:45:21 dist-test-slave-1nl6 krb5kdc[1507](info): set up 2 sockets
Aug 10 00:45:21 dist-test-slave-1nl6 krb5kdc[1507](info): commencing operation
krb5kdc: starting...
W20260810 00:45:22.741911 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.246s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:45:22 dist-test-slave-1nl6 krb5kdc[1507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:45:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:45:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.6.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
I20260810 00:45:22.943929 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:33943
--webserver_interface=127.28.6.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:46599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:33943
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:45:23.265885 1526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:23.266161 1526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:23.266249 1526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:23.273120 1526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:45:23.273216 1526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:23.273273 1526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:23.273335 1526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:45:23.273380 1526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:45:23.282485 1526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:46599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:33943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:33943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:23.283615 1526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:23.284803 1526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:23.296530 1532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:23.296483 1534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:23.296748 1536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:23.297148 1526 server_base.cc:1034] running on GCE node
I20260810 00:45:23.297899 1526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:23.299187 1526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:23.300395 1526 hybrid_clock.cc:648] HybridClock initialized: now 1786322723300356 us; error 24 us; skew 500 ppm
I20260810 00:45:23.300747 1526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:45:23 dist-test-slave-1nl6 krb5kdc[1507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322723, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:23.307927 1526 init.cc:377] Logged in from keytab as kudu/127.28.6.62@KRBTEST.COM (short username kudu)
I20260810 00:45:23.309824 1526 webserver.cc:460] Webserver started at http://127.28.6.62:38131/ using document root <none> and password file <none>
I20260810 00:45:23.310551 1526 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:23.310698 1526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:23.311115 1526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:23.314082 1526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "6c7c8e7b465e4194b847ff28469e9a5d"
format_stamp: "Formatted at 2026-08-10 00:45:23 on dist-test-slave-1nl6"
server_key: "53ebe7b25698eb85eb44a7ddd8e87c4b"
server_key_iv: "0013ba9b25b33c2f47d2461fb35615f1"
server_key_version: "encryptionkey@0"
I20260810 00:45:23.314888 1526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "6c7c8e7b465e4194b847ff28469e9a5d"
format_stamp: "Formatted at 2026-08-10 00:45:23 on dist-test-slave-1nl6"
server_key: "53ebe7b25698eb85eb44a7ddd8e87c4b"
server_key_iv: "0013ba9b25b33c2f47d2461fb35615f1"
server_key_version: "encryptionkey@0"
I20260810 00:45:23.319976 1526 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 00:45:23.323341 1548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:23.324609 1526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 00:45:23.324803 1526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "6c7c8e7b465e4194b847ff28469e9a5d"
format_stamp: "Formatted at 2026-08-10 00:45:23 on dist-test-slave-1nl6"
server_key: "53ebe7b25698eb85eb44a7ddd8e87c4b"
server_key_iv: "0013ba9b25b33c2f47d2461fb35615f1"
server_key_version: "encryptionkey@0"
I20260810 00:45:23.324999 1526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:23.338644 1526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:23.342548 1526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:23.342831 1526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:23.361426 1526 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:33943
I20260810 00:45:23.361474 1621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:33943 every 8 connection(s)
I20260810 00:45:23.362782 1526 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:45:23.366290 1622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:23.372117 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 1526
I20260810 00:45:23.372354 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:45:23.372983 28696 external_mini_cluster.cc:1428] Setting key 79c1cd987cb2c1afc16e8df7f2c25661
I20260810 00:45:23.376762 1622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c7c8e7b465e4194b847ff28469e9a5d: Bootstrap starting.
I20260810 00:45:23.380374 1622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c7c8e7b465e4194b847ff28469e9a5d: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:23.381405 1622 log.cc:824] T 00000000000000000000000000000000 P 6c7c8e7b465e4194b847ff28469e9a5d: Log is configured to *not* fsync() on all Append() calls
I20260810 00:45:23.384413 1622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c7c8e7b465e4194b847ff28469e9a5d: No bootstrap required, opened a new log
Aug 10 00:45:23 dist-test-slave-1nl6 krb5kdc[1507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:45:23.395104 1622 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c7c8e7b465e4194b847ff28469e9a5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c7c8e7b465e4194b847ff28469e9a5d" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 33943 } }
I20260810 00:45:23.395574 1622 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c7c8e7b465e4194b847ff28469e9a5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:23.395720 1622 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c7c8e7b465e4194b847ff28469e9a5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c7c8e7b465e4194b847ff28469e9a5d, State: Initialized, Role: FOLLOWER
I20260810 00:45:23.396385 1622 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c7c8e7b465e4194b847ff28469e9a5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c7c8e7b465e4194b847ff28469e9a5d" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 33943 } }
I20260810 00:45:23.396689 1622 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c7c8e7b465e4194b847ff28469e9a5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:45:23.396847 1622 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c7c8e7b465e4194b847ff28469e9a5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:45:23.397042 1622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c7c8e7b465e4194b847ff28469e9a5d [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:23.397346 1626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:23.375013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54414 (local address 127.28.6.62:33943)
0810 00:45:23.375776 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:23.375788 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:23.375827 (+ 39us) server_negotiation.cc:407] Connection header received
0810 00:45:23.377539 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:23.377562 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:23.377910 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:23.378219 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:23.380094 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:23.381449 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:23.383079 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:23.383712 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:23.383841 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:23.388556 (+ 4715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:23.388568 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:23.388593 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:23.388640 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:23.392611 (+ 3971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:23.393293 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:23.393302 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:23.393311 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:23.393532 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:23.393960 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:23.393967 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:23.393975 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:23.394686 (+ 711us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:23.395031 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:23.395474 (+ 443us) server_negotiation.cc:299] Negotiation successful
0810 00:45:23.395926 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":353,"threads_started":1}
I20260810 00:45:23.399545 1622 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c7c8e7b465e4194b847ff28469e9a5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c7c8e7b465e4194b847ff28469e9a5d" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 33943 } }
I20260810 00:45:23.400051 1622 leader_election.cc:302] T 00000000000000000000000000000000 P 6c7c8e7b465e4194b847ff28469e9a5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c7c8e7b465e4194b847ff28469e9a5d; no voters:
I20260810 00:45:23.400611 1622 leader_election.cc:288] T 00000000000000000000000000000000 P 6c7c8e7b465e4194b847ff28469e9a5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:45:23.400738 1628 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c7c8e7b465e4194b847ff28469e9a5d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:45:23.401428 1628 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c7c8e7b465e4194b847ff28469e9a5d [term 1 LEADER]: Becoming Leader. State: Replica: 6c7c8e7b465e4194b847ff28469e9a5d, State: Running, Role: LEADER
I20260810 00:45:23.401922 1628 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c7c8e7b465e4194b847ff28469e9a5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c7c8e7b465e4194b847ff28469e9a5d" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 33943 } }
I20260810 00:45:23.402637 1622 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c7c8e7b465e4194b847ff28469e9a5d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:45:23.404824 1631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c7c8e7b465e4194b847ff28469e9a5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c7c8e7b465e4194b847ff28469e9a5d. Latest consensus state: current_term: 1 leader_uuid: "6c7c8e7b465e4194b847ff28469e9a5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c7c8e7b465e4194b847ff28469e9a5d" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 33943 } } }
I20260810 00:45:23.405282 1631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c7c8e7b465e4194b847ff28469e9a5d [sys.catalog]: This master's current role is: LEADER
I20260810 00:45:23.405107 1629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c7c8e7b465e4194b847ff28469e9a5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c7c8e7b465e4194b847ff28469e9a5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c7c8e7b465e4194b847ff28469e9a5d" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 33943 } } }
I20260810 00:45:23.405846 1629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c7c8e7b465e4194b847ff28469e9a5d [sys.catalog]: This master's current role is: LEADER
I20260810 00:45:23.408004 1644 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:45:23.413148 1644 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:45:23.421824 1644 catalog_manager.cc:1347] Generated new cluster ID: 1ba10e2685214fa1bacf68a0e21fa577
I20260810 00:45:23.421931 1644 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:45:23.430028 1644 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:45:23.431752 1644 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:45:23.446498 1644 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c7c8e7b465e4194b847ff28469e9a5d: Generated new TSK 0
I20260810 00:45:23.447310 1644 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.6.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab.
Entry for principal kudu/127.28.6.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab.
I20260810 00:45:23.504153 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.1:0
--local_ip_for_outbound_sockets=127.28.6.1
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:33943
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:46599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:45:23.856405 1660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:23.856700 1660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:23.856892 1660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:23.863721 1660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:23.863850 1660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:23.864023 1660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.1
I20260810 00:45:23.873108 1660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:46599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:33943
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.1
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:23.874363 1660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:23.875658 1660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:23.889294 1666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:23.889391 1667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:23.889456 1670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:23.890398 1660 server_base.cc:1034] running on GCE node
I20260810 00:45:23.891122 1660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:23.892094 1660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:23.893352 1660 hybrid_clock.cc:648] HybridClock initialized: now 1786322723893297 us; error 44 us; skew 500 ppm
I20260810 00:45:23.893738 1660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:45:23 dist-test-slave-1nl6 krb5kdc[1507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322723, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:23.897152 1660 init.cc:377] Logged in from keytab as kudu/127.28.6.1@KRBTEST.COM (short username kudu)
I20260810 00:45:23.898552 1660 webserver.cc:460] Webserver started at http://127.28.6.1:44075/ using document root <none> and password file <none>
I20260810 00:45:23.899256 1660 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:23.899364 1660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:23.899621 1660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:23.902387 1660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-0/data/instance:
uuid: "dfe5a265c4ba49bf9def48881640ae98"
format_stamp: "Formatted at 2026-08-10 00:45:23 on dist-test-slave-1nl6"
server_key: "dc511b808e7a25af29ad915c14158cf0"
server_key_iv: "4cbd708aa18278ceaea6e805d128dd6d"
server_key_version: "encryptionkey@0"
I20260810 00:45:23.903316 1660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance:
uuid: "dfe5a265c4ba49bf9def48881640ae98"
format_stamp: "Formatted at 2026-08-10 00:45:23 on dist-test-slave-1nl6"
server_key: "dc511b808e7a25af29ad915c14158cf0"
server_key_iv: "4cbd708aa18278ceaea6e805d128dd6d"
server_key_version: "encryptionkey@0"
I20260810 00:45:23.908349 1660 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260810 00:45:23.911738 1684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:23.913048 1660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260810 00:45:23.913242 1660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-0/wal
uuid: "dfe5a265c4ba49bf9def48881640ae98"
format_stamp: "Formatted at 2026-08-10 00:45:23 on dist-test-slave-1nl6"
server_key: "dc511b808e7a25af29ad915c14158cf0"
server_key_iv: "4cbd708aa18278ceaea6e805d128dd6d"
server_key_version: "encryptionkey@0"
I20260810 00:45:23.913448 1660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:23.928732 1660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:23.933168 1660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:23.933463 1660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:23.934379 1660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:45:23.935660 1660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:45:23.935786 1660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:23.935936 1660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:45:23.936017 1660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:23.969058 1660 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.1:43217
I20260810 00:45:23.969146 1841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.1:43217 every 8 connection(s)
I20260810 00:45:23.970373 1660 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
I20260810 00:45:23.975471 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 1660
I20260810 00:45:23.975709 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance
I20260810 00:45:23.976341 28696 external_mini_cluster.cc:1428] Setting key f67b31aaa4500f850387bb763e3fa6da
Aug 10 00:45:23 dist-test-slave-1nl6 krb5kdc[1507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322723, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:45:23.992877 1844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:23.972929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:49821 (local address 127.28.6.62:33943)
0810 00:45:23.973338 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:23.973342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:23.974307 (+ 965us) server_negotiation.cc:407] Connection header received
0810 00:45:23.976281 (+ 1974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:23.976287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:23.976383 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:23.976512 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:23.978397 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:23.979156 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:23.981722 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:23.982106 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:23.982194 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:23.986336 (+ 4142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:23.986341 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:23.986345 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:23.986377 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:23.989521 (+ 3144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:23.990230 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:23.990238 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:23.990244 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:23.990401 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:23.990830 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:23.990838 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:23.990845 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:23.991239 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:23.991395 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:23.992100 (+ 705us) server_negotiation.cc:299] Negotiation successful
0810 00:45:23.992380 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":198,"threads_started":1}
I20260810 00:45:23.993395 1845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:23.973434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:33943 (local address 127.28.6.1:49821)
0810 00:45:23.974152 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:23.974196 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:23.976012 (+ 1816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:23.976826 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:23.976837 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:23.977536 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:23.978184 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:23.978209 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:23.979330 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:23.979339 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:23.981565 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:23.981585 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:23.982324 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:23.982335 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:23.982679 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:23.983436 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:23.983469 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:23.986108 (+ 2639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:23.989671 (+ 3563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:23.989677 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:23.989689 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:23.990098 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:23.990515 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:23.990518 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:23.990521 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:23.990699 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:23.991367 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:23.991373 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:23.991673 (+ 300us) client_negotiation.cc:769] Sending connection context
0810 00:45:23.991910 (+ 237us) client_negotiation.cc:240] Negotiation successful
0810 00:45:23.992289 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":274,"threads_started":1}
I20260810 00:45:23.994980 1842 heartbeater.cc:347] Connected to a master server at 127.28.6.62:33943
I20260810 00:45:23.995308 1842 heartbeater.cc:464] Registering TS with master...
I20260810 00:45:23.996234 1842 heartbeater.cc:511] Master 127.28.6.62:33943 requested a full tablet report, sending...
I20260810 00:45:23.998508 1570 ts_manager.cc:194] Registered new tserver with Master: dfe5a265c4ba49bf9def48881640ae98 (127.28.6.1:43217)
WARNING: no policy specified for kudu/127.28.6.2@KRBTEST.COM; defaulting to no policy
I20260810 00:45:24.000037 1570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.1@KRBTEST.COM'} at 127.28.6.1:49821
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab.
Entry for principal kudu/127.28.6.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab.
I20260810 00:45:24.025931 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.2:0
--local_ip_for_outbound_sockets=127.28.6.2
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:33943
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:46599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:45:24.356170 1849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:24.356499 1849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:24.356676 1849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:24.365397 1849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:24.365521 1849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:24.365701 1849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.2
I20260810 00:45:24.381036 1849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:46599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:33943
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.2
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:24.382588 1849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:24.384275 1849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:24.397727 1855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:24.397763 1856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:24.397728 1858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:24.399034 1849 server_base.cc:1034] running on GCE node
I20260810 00:45:24.399624 1849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:24.400535 1849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:24.401764 1849 hybrid_clock.cc:648] HybridClock initialized: now 1786322724401721 us; error 72 us; skew 500 ppm
I20260810 00:45:24.402071 1849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:45:24 dist-test-slave-1nl6 krb5kdc[1507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322724, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:24.405365 1849 init.cc:377] Logged in from keytab as kudu/127.28.6.2@KRBTEST.COM (short username kudu)
I20260810 00:45:24.406808 1849 webserver.cc:460] Webserver started at http://127.28.6.2:46263/ using document root <none> and password file <none>
I20260810 00:45:24.407588 1849 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:24.407739 1849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:24.408104 1849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:24.411126 1849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-1/data/instance:
uuid: "b364c756f86e4868a11cf5bb0720d4eb"
format_stamp: "Formatted at 2026-08-10 00:45:24 on dist-test-slave-1nl6"
server_key: "3e5834faf1d0c65b02b18435d257686e"
server_key_iv: "7698e97f6e90334d40b4cf938bbfc9e2"
server_key_version: "encryptionkey@0"
I20260810 00:45:24.411954 1849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance:
uuid: "b364c756f86e4868a11cf5bb0720d4eb"
format_stamp: "Formatted at 2026-08-10 00:45:24 on dist-test-slave-1nl6"
server_key: "3e5834faf1d0c65b02b18435d257686e"
server_key_iv: "7698e97f6e90334d40b4cf938bbfc9e2"
server_key_version: "encryptionkey@0"
I20260810 00:45:24.416872 1849 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 00:45:24.420414 1865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:24.421608 1849 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 00:45:24.421784 1849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-1/wal
uuid: "b364c756f86e4868a11cf5bb0720d4eb"
format_stamp: "Formatted at 2026-08-10 00:45:24 on dist-test-slave-1nl6"
server_key: "3e5834faf1d0c65b02b18435d257686e"
server_key_iv: "7698e97f6e90334d40b4cf938bbfc9e2"
server_key_version: "encryptionkey@0"
I20260810 00:45:24.421983 1849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:24.436589 1849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:24.441741 1849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:24.442034 1849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:24.443007 1849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:45:24.444314 1849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:45:24.444408 1849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:24.444588 1849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:45:24.444641 1849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:24.478374 1849 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.2:42785
I20260810 00:45:24.478417 2031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.2:42785 every 8 connection(s)
I20260810 00:45:24.479669 1849 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
I20260810 00:45:24.484589 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 1849
I20260810 00:45:24.484789 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance
I20260810 00:45:24.485216 28696 external_mini_cluster.cc:1428] Setting key 14721ed0dbfaec71289bae1ff87d4244
Aug 10 00:45:24 dist-test-slave-1nl6 krb5kdc[1507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322724, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:45:24.502725 1844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:24.482225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:50023 (local address 127.28.6.62:33943)
0810 00:45:24.482411 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:24.482416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:24.483565 (+ 1149us) server_negotiation.cc:407] Connection header received
0810 00:45:24.485527 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:24.485533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:24.485625 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:24.485748 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:24.487525 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:24.488216 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:24.490691 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:24.491166 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:24.491258 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:24.495620 (+ 4362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:24.495629 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:24.495636 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:24.495680 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:24.499406 (+ 3726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:24.500094 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:24.500102 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:24.500109 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:24.500278 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:24.500734 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:24.500741 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:24.500746 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:24.501155 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:24.501313 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:24.501939 (+ 626us) server_negotiation.cc:299] Negotiation successful
0810 00:45:24.502232 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 00:45:24.503333 2034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:24.482639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:33943 (local address 127.28.6.2:50023)
0810 00:45:24.483407 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:24.483446 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:24.485230 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:24.485996 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:24.486009 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:24.486573 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:24.487326 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:24.487352 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:24.488400 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:24.488409 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:24.490552 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:24.490570 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:24.491395 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:24.491406 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:24.491733 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:24.492461 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:24.492496 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:24.495399 (+ 2903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:24.499536 (+ 4137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:24.499542 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:24.499552 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:24.499958 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:24.500400 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:24.500403 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:24.500405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:24.500589 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:24.501278 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:24.501283 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:24.501579 (+ 296us) client_negotiation.cc:769] Sending connection context
0810 00:45:24.501797 (+ 218us) client_negotiation.cc:240] Negotiation successful
0810 00:45:24.502140 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":331,"threads_started":1}
I20260810 00:45:24.504890 2032 heartbeater.cc:347] Connected to a master server at 127.28.6.62:33943
I20260810 00:45:24.505196 2032 heartbeater.cc:464] Registering TS with master...
I20260810 00:45:24.505903 2032 heartbeater.cc:511] Master 127.28.6.62:33943 requested a full tablet report, sending...
I20260810 00:45:24.507562 1570 ts_manager.cc:194] Registered new tserver with Master: b364c756f86e4868a11cf5bb0720d4eb (127.28.6.2:42785)
I20260810 00:45:24.508798 1570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.2@KRBTEST.COM'} at 127.28.6.2:50023
WARNING: no policy specified for kudu/127.28.6.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab.
Entry for principal kudu/127.28.6.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab.
I20260810 00:45:24.534186 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.3:0
--local_ip_for_outbound_sockets=127.28.6.3
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:33943
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:46599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:45:24.862632 2038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:24.862999 2038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:24.863171 2038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:24.870319 2038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:24.870469 2038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:24.870652 2038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.3
I20260810 00:45:24.880149 2038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:46599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:33943
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.3
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:24.881390 2038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:24.882711 2038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:24.895804 2052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:24.895903 2047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:24.895872 2046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:24.900345 2038 server_base.cc:1034] running on GCE node
I20260810 00:45:24.900936 2038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:24.901829 2038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:24.903112 2038 hybrid_clock.cc:648] HybridClock initialized: now 1786322724903057 us; error 54 us; skew 500 ppm
I20260810 00:45:24.903450 2038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:45:24 dist-test-slave-1nl6 krb5kdc[1507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322724, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:24.906625 2038 init.cc:377] Logged in from keytab as kudu/127.28.6.3@KRBTEST.COM (short username kudu)
I20260810 00:45:24.908264 2038 webserver.cc:460] Webserver started at http://127.28.6.3:40801/ using document root <none> and password file <none>
I20260810 00:45:24.908900 2038 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:24.909000 2038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:24.909255 2038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:24.912123 2038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-2/data/instance:
uuid: "0a67c1f7c40441d7b13ffb251eeb5040"
format_stamp: "Formatted at 2026-08-10 00:45:24 on dist-test-slave-1nl6"
server_key: "82f67124e9b171565541f6f36964d753"
server_key_iv: "1b870ed220ae5a339ae33ebe101df76b"
server_key_version: "encryptionkey@0"
I20260810 00:45:24.912878 2038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance:
uuid: "0a67c1f7c40441d7b13ffb251eeb5040"
format_stamp: "Formatted at 2026-08-10 00:45:24 on dist-test-slave-1nl6"
server_key: "82f67124e9b171565541f6f36964d753"
server_key_iv: "1b870ed220ae5a339ae33ebe101df76b"
server_key_version: "encryptionkey@0"
I20260810 00:45:24.917856 2038 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260810 00:45:24.921131 2061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:24.922371 2038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 00:45:24.922544 2038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-2/wal
uuid: "0a67c1f7c40441d7b13ffb251eeb5040"
format_stamp: "Formatted at 2026-08-10 00:45:24 on dist-test-slave-1nl6"
server_key: "82f67124e9b171565541f6f36964d753"
server_key_iv: "1b870ed220ae5a339ae33ebe101df76b"
server_key_version: "encryptionkey@0"
I20260810 00:45:24.922719 2038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:24.938360 2038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:24.942041 2038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:24.942337 2038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:24.943339 2038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:45:24.944622 2038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:45:24.944695 2038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:24.944818 2038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:45:24.944882 2038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:24.978860 2038 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.3:41493
I20260810 00:45:24.978977 2220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.3:41493 every 8 connection(s)
I20260810 00:45:24.980154 2038 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
I20260810 00:45:24.985975 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 2038
I20260810 00:45:24.986209 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance
I20260810 00:45:24.986678 28696 external_mini_cluster.cc:1428] Setting key a8dc5b0ec39b5b7c7f6bdcd9434efd79
Aug 10 00:45:24 dist-test-slave-1nl6 krb5kdc[1507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322724, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:45:24.999615 1844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:24.982748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:35191 (local address 127.28.6.62:33943)
0810 00:45:24.982984 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:24.982989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:24.984064 (+ 1075us) server_negotiation.cc:407] Connection header received
0810 00:45:24.985850 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:24.985856 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:24.985947 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:24.986080 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:24.987796 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:24.988778 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:24.990328 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:24.990721 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:24.990859 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:24.994163 (+ 3304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:24.994168 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:24.994171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:24.994203 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:24.996640 (+ 2437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:24.997336 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:24.997347 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:24.997350 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:24.997485 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:24.997946 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:24.997950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:24.997952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:24.998205 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:24.998364 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:24.999049 (+ 685us) server_negotiation.cc:299] Negotiation successful
0810 00:45:24.999269 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 00:45:25.000473 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:24.983225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:33943 (local address 127.28.6.3:35191)
0810 00:45:24.983904 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:24.983946 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:24.985581 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:24.986346 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:24.986359 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:24.986946 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:24.987627 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:24.987646 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:24.988957 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:24.988960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:24.990166 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:24.990178 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:24.990960 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:24.990971 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:24.991245 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:24.991840 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:24.991862 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:24.993941 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:24.996781 (+ 2840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:24.996790 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:24.996804 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:24.997205 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:24.997600 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:24.997605 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:24.997608 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:24.997813 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:24.998346 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:24.998353 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:24.998656 (+ 303us) client_negotiation.cc:769] Sending connection context
0810 00:45:24.998899 (+ 243us) client_negotiation.cc:240] Negotiation successful
0810 00:45:24.999426 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":256,"threads_started":1}
I20260810 00:45:25.001910 2221 heartbeater.cc:347] Connected to a master server at 127.28.6.62:33943
I20260810 00:45:25.002233 2221 heartbeater.cc:464] Registering TS with master...
I20260810 00:45:25.002977 1842 heartbeater.cc:503] Master 127.28.6.62:33943 was elected leader, sending a full tablet report...
I20260810 00:45:25.003070 2221 heartbeater.cc:511] Master 127.28.6.62:33943 requested a full tablet report, sending...
I20260810 00:45:25.004719 1570 ts_manager.cc:194] Registered new tserver with Master: 0a67c1f7c40441d7b13ffb251eeb5040 (127.28.6.3:41493)
I20260810 00:45:25.005475 1570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.3@KRBTEST.COM'} at 127.28.6.3:35191
I20260810 00:45:25.013995 28696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 00:45:25 dist-test-slave-1nl6 krb5kdc[1507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 00:45:25 dist-test-slave-1nl6 krb5kdc[1507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.1@KRBTEST.COM
I20260810 00:45:25.066349 2231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:25.051408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35134 (local address 127.28.6.1:43217)
0810 00:45:25.052073 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:25.052082 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:25.052109 (+ 27us) server_negotiation.cc:407] Connection header received
0810 00:45:25.052186 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:25.052193 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:25.052414 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:25.052542 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:25.053881 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:25.054990 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:25.056169 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:25.056738 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:25.056834 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:25.061112 (+ 4278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:25.061124 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:25.061137 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:25.061169 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:25.063802 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:25.064513 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:25.064520 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:25.064522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:25.064635 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:25.065042 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:25.065045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:25.065047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:25.065373 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:25.065592 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:25.065868 (+ 276us) server_negotiation.cc:299] Negotiation successful
0810 00:45:25.066055 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":339,"threads_started":1}
Aug 10 00:45:25 dist-test-slave-1nl6 krb5kdc[1507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:45:25.083518 1844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:25.070704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54424 (local address 127.28.6.62:33943)
0810 00:45:25.070946 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:25.070953 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:25.071157 (+ 204us) server_negotiation.cc:407] Connection header received
0810 00:45:25.071351 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:25.071357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:25.071429 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:25.071529 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:25.072730 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:25.073423 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:25.074627 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:25.075099 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:25.075229 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:25.078845 (+ 3616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:25.078850 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:25.078853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:25.078884 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:25.081326 (+ 2442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:25.081887 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:25.081890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:25.081893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:25.082005 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:25.082296 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:25.082299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:25.082301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:25.082612 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:25.082785 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:25.083109 (+ 324us) server_negotiation.cc:299] Negotiation successful
0810 00:45:25.083276 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260810 00:45:25.084092 1570 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54424
I20260810 00:45:25.085129 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 1660
I20260810 00:45:25.097410 2231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:25.086017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35150 (local address 127.28.6.1:43217)
0810 00:45:25.086175 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:25.086179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:25.086421 (+ 242us) server_negotiation.cc:407] Connection header received
0810 00:45:25.086538 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:25.086541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:25.086600 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:25.086675 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:25.087825 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:25.088449 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:25.089595 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:25.090247 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:25.090378 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:25.092415 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:25.092420 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:25.092423 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:25.092453 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:25.095411 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:25.095979 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:25.095983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:25.095985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:25.096101 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:25.096390 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:25.096394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:25.096396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:25.096609 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:25.096784 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:25.097019 (+ 235us) server_negotiation.cc:299] Negotiation successful
0810 00:45:25.097165 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 00:45:25.098018 1835 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:45:25.179803 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 1660
I20260810 00:45:25.203388 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 1849
Aug 10 00:45:25 dist-test-slave-1nl6 krb5kdc[1507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.2@KRBTEST.COM
I20260810 00:45:25.219501 2239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:25.204311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50880 (local address 127.28.6.2:42785)
0810 00:45:25.204890 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:25.204897 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:25.204923 (+ 26us) server_negotiation.cc:407] Connection header received
0810 00:45:25.204989 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:25.204996 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:25.205228 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:25.205420 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:25.206695 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:25.207874 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:25.209151 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:25.209717 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:25.209860 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:25.214030 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:25.214042 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:25.214057 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:25.214092 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:25.216882 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:25.217521 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:25.217528 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:25.217531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:25.217655 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:25.217988 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:25.217991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:25.217993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:25.218355 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:25.218576 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:25.218829 (+ 253us) server_negotiation.cc:299] Negotiation successful
0810 00:45:25.219089 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":272,"threads_started":1}
I20260810 00:45:25.220806 2026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:45:25.301155 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 1849
I20260810 00:45:25.328902 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 2038
Aug 10 00:45:25 dist-test-slave-1nl6 krb5kdc[1507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.3@KRBTEST.COM
I20260810 00:45:25.345057 2241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:25.330004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35298 (local address 127.28.6.3:41493)
0810 00:45:25.330626 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:25.330633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:25.330687 (+ 54us) server_negotiation.cc:407] Connection header received
0810 00:45:25.330750 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:25.330756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:25.331041 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:25.331223 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:25.332471 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:25.333494 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:25.334872 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:25.335497 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:25.335596 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:25.339578 (+ 3982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:25.339590 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:25.339604 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:25.339636 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:25.342540 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:25.343235 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:25.343242 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:25.343244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:25.343364 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:25.343718 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:25.343722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:25.343724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:25.344066 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:25.344301 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:25.344562 (+ 261us) server_negotiation.cc:299] Negotiation successful
0810 00:45:25.344774 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":321,"threads_started":1}
I20260810 00:45:25.346410 2215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:45:25.429315 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 2038
I20260810 00:45:25.455139 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 1526
I20260810 00:45:25.456364 1617 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:45:25.558072 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 1526
2026-08-10T00:45:25Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4114 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:45:25 dist-test-slave-1nl6 krb5kdc[2248](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:45:25 dist-test-slave-1nl6 krb5kdc[2248](info): set up 2 sockets
Aug 10 00:45:25 dist-test-slave-1nl6 krb5kdc[2248](info): commencing operation
krb5kdc: starting...
W20260810 00:45:26.856180 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.247s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:45:26 dist-test-slave-1nl6 krb5kdc[2248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:45:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:45:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.6.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
I20260810 00:45:27.059583 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:45301
--webserver_interface=127.28.6.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:41179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:45301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:45:27.381039 2265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:27.381311 2265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:27.381510 2265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:27.388499 2265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:45:27.388589 2265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:27.388693 2265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:27.388794 2265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:45:27.388901 2265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:45:27.398300 2265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:41179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:45301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:45301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:27.399502 2265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:27.400759 2265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:27.412580 2279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:27.412654 2270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:27.412693 2276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:27.413544 2265 server_base.cc:1034] running on GCE node
I20260810 00:45:27.414296 2265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:27.415630 2265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:27.416883 2265 hybrid_clock.cc:648] HybridClock initialized: now 1786322727416854 us; error 57 us; skew 500 ppm
I20260810 00:45:27.417259 2265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:45:27 dist-test-slave-1nl6 krb5kdc[2248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322727, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:27.420466 2265 init.cc:377] Logged in from keytab as kudu/127.28.6.62@KRBTEST.COM (short username kudu)
I20260810 00:45:27.421641 2265 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:27.421785 2265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:27.422111 2265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:27.424988 2265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "79155b0a285b4a9b86dbe7094f3f0115"
format_stamp: "Formatted at 2026-08-10 00:45:27 on dist-test-slave-1nl6"
server_key: "1a1e974545095cf961bef26bcc4a3584"
server_key_iv: "69fbab2a8b054c72e47c6b400f886b21"
server_key_version: "encryptionkey@0"
I20260810 00:45:27.425792 2265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "79155b0a285b4a9b86dbe7094f3f0115"
format_stamp: "Formatted at 2026-08-10 00:45:27 on dist-test-slave-1nl6"
server_key: "1a1e974545095cf961bef26bcc4a3584"
server_key_iv: "69fbab2a8b054c72e47c6b400f886b21"
server_key_version: "encryptionkey@0"
I20260810 00:45:27.430711 2265 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260810 00:45:27.434194 2285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:27.435523 2265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 00:45:27.435719 2265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "79155b0a285b4a9b86dbe7094f3f0115"
format_stamp: "Formatted at 2026-08-10 00:45:27 on dist-test-slave-1nl6"
server_key: "1a1e974545095cf961bef26bcc4a3584"
server_key_iv: "69fbab2a8b054c72e47c6b400f886b21"
server_key_version: "encryptionkey@0"
I20260810 00:45:27.435947 2265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:27.461473 2265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:27.465718 2265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:27.465984 2265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:27.484025 2265 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:45301
I20260810 00:45:27.484047 2369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:45301 every 8 connection(s)
I20260810 00:45:27.484897 2265 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:45:27.487659 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 2265
I20260810 00:45:27.487905 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:45:27.488737 28696 external_mini_cluster.cc:1428] Setting key 3034bd6f6f2376d34b94d841e6601fae
I20260810 00:45:27.489017 2370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:27.503403 2370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79155b0a285b4a9b86dbe7094f3f0115: Bootstrap starting.
Aug 10 00:45:27 dist-test-slave-1nl6 krb5kdc[2248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:45:27.506489 2370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79155b0a285b4a9b86dbe7094f3f0115: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:27.507619 2370 log.cc:824] T 00000000000000000000000000000000 P 79155b0a285b4a9b86dbe7094f3f0115: Log is configured to *not* fsync() on all Append() calls
I20260810 00:45:27.510314 2370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79155b0a285b4a9b86dbe7094f3f0115: No bootstrap required, opened a new log
I20260810 00:45:27.511917 2377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:27.490786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53034 (local address 127.28.6.62:45301)
0810 00:45:27.491624 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:27.491638 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:27.491682 (+ 44us) server_negotiation.cc:407] Connection header received
0810 00:45:27.493600 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:27.493624 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:27.493988 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:27.494295 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:27.496202 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:27.497468 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:27.499361 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:27.500056 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:27.500185 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:27.504735 (+ 4550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:27.504743 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:27.504759 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:27.504796 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:27.507885 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:27.508652 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:27.508658 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:27.508664 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:27.508837 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:27.509269 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:27.509273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:27.509276 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:27.509817 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:27.510136 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:27.510581 (+ 445us) server_negotiation.cc:299] Negotiation successful
0810 00:45:27.511026 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":442,"threads_started":1}
I20260810 00:45:27.517962 2370 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79155b0a285b4a9b86dbe7094f3f0115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79155b0a285b4a9b86dbe7094f3f0115" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 45301 } }
I20260810 00:45:27.518424 2370 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79155b0a285b4a9b86dbe7094f3f0115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:27.518549 2370 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79155b0a285b4a9b86dbe7094f3f0115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79155b0a285b4a9b86dbe7094f3f0115, State: Initialized, Role: FOLLOWER
I20260810 00:45:27.519207 2370 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79155b0a285b4a9b86dbe7094f3f0115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79155b0a285b4a9b86dbe7094f3f0115" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 45301 } }
I20260810 00:45:27.519457 2370 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79155b0a285b4a9b86dbe7094f3f0115 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:45:27.519600 2370 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79155b0a285b4a9b86dbe7094f3f0115 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:45:27.519785 2370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79155b0a285b4a9b86dbe7094f3f0115 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:27.521899 2370 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79155b0a285b4a9b86dbe7094f3f0115 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79155b0a285b4a9b86dbe7094f3f0115" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 45301 } }
I20260810 00:45:27.522387 2370 leader_election.cc:302] T 00000000000000000000000000000000 P 79155b0a285b4a9b86dbe7094f3f0115 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79155b0a285b4a9b86dbe7094f3f0115; no voters:
I20260810 00:45:27.523028 2370 leader_election.cc:288] T 00000000000000000000000000000000 P 79155b0a285b4a9b86dbe7094f3f0115 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:45:27.523267 2379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79155b0a285b4a9b86dbe7094f3f0115 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:45:27.523993 2379 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79155b0a285b4a9b86dbe7094f3f0115 [term 1 LEADER]: Becoming Leader. State: Replica: 79155b0a285b4a9b86dbe7094f3f0115, State: Running, Role: LEADER
I20260810 00:45:27.524500 2379 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79155b0a285b4a9b86dbe7094f3f0115 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79155b0a285b4a9b86dbe7094f3f0115" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 45301 } }
I20260810 00:45:27.525118 2370 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79155b0a285b4a9b86dbe7094f3f0115 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:45:27.527351 2380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79155b0a285b4a9b86dbe7094f3f0115 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79155b0a285b4a9b86dbe7094f3f0115" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79155b0a285b4a9b86dbe7094f3f0115" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 45301 } } }
I20260810 00:45:27.527709 2380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79155b0a285b4a9b86dbe7094f3f0115 [sys.catalog]: This master's current role is: LEADER
I20260810 00:45:27.529218 2383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79155b0a285b4a9b86dbe7094f3f0115 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79155b0a285b4a9b86dbe7094f3f0115. Latest consensus state: current_term: 1 leader_uuid: "79155b0a285b4a9b86dbe7094f3f0115" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79155b0a285b4a9b86dbe7094f3f0115" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 45301 } } }
I20260810 00:45:27.529493 2383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79155b0a285b4a9b86dbe7094f3f0115 [sys.catalog]: This master's current role is: LEADER
I20260810 00:45:27.530086 2392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:45:27.535370 2392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:45:27.544538 2392 catalog_manager.cc:1347] Generated new cluster ID: ff1d21a7f4074e68a5c06a1395748507
I20260810 00:45:27.544700 2392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:45:27.560488 2392 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:45:27.562330 2392 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:45:27.569024 2392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79155b0a285b4a9b86dbe7094f3f0115: Generated new TSK 0
I20260810 00:45:27.569973 2392 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.6.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab.
Entry for principal kudu/127.28.6.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab.
I20260810 00:45:27.626063 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.1:0
--local_ip_for_outbound_sockets=127.28.6.1
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:45301
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:41179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:45:27.954797 2411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:27.955073 2411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:27.955251 2411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:27.962186 2411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:27.962283 2411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:27.962456 2411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.1
I20260810 00:45:27.971915 2411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:41179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:45301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.1
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:27.973137 2411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:27.974447 2411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:27.987210 2421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:27.987246 2425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:27.987282 2423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:27.987864 2411 server_base.cc:1034] running on GCE node
I20260810 00:45:27.988440 2411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:27.989331 2411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:27.990557 2411 hybrid_clock.cc:648] HybridClock initialized: now 1786322727990511 us; error 60 us; skew 500 ppm
I20260810 00:45:27.990867 2411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:45:27 dist-test-slave-1nl6 krb5kdc[2248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322727, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:28.003388 2411 init.cc:377] Logged in from keytab as kudu/127.28.6.1@KRBTEST.COM (short username kudu)
I20260810 00:45:28.004619 2411 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:28.004753 2411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:28.005103 2411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:28.008172 2411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-0/data/instance:
uuid: "26fd688e3b6244deab1ce974911358c0"
format_stamp: "Formatted at 2026-08-10 00:45:27 on dist-test-slave-1nl6"
server_key: "ad3236e29ecb3dbbe714eade807b7c71"
server_key_iv: "6cc2c27872017fdf41d1b33f6beec0f8"
server_key_version: "encryptionkey@0"
I20260810 00:45:28.009151 2411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance:
uuid: "26fd688e3b6244deab1ce974911358c0"
format_stamp: "Formatted at 2026-08-10 00:45:27 on dist-test-slave-1nl6"
server_key: "ad3236e29ecb3dbbe714eade807b7c71"
server_key_iv: "6cc2c27872017fdf41d1b33f6beec0f8"
server_key_version: "encryptionkey@0"
I20260810 00:45:28.014394 2411 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260810 00:45:28.018007 2434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:28.019375 2411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 00:45:28.019560 2411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-0/wal
uuid: "26fd688e3b6244deab1ce974911358c0"
format_stamp: "Formatted at 2026-08-10 00:45:27 on dist-test-slave-1nl6"
server_key: "ad3236e29ecb3dbbe714eade807b7c71"
server_key_iv: "6cc2c27872017fdf41d1b33f6beec0f8"
server_key_version: "encryptionkey@0"
I20260810 00:45:28.019795 2411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:28.043673 2411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:28.047586 2411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:28.047868 2411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:28.048880 2411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:45:28.050186 2411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:45:28.050314 2411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:28.050434 2411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:45:28.050521 2411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:28.083793 2411 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.1:42751
I20260810 00:45:28.083880 2593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.1:42751 every 8 connection(s)
I20260810 00:45:28.084672 2411 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
I20260810 00:45:28.087740 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 2411
I20260810 00:45:28.087961 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance
I20260810 00:45:28.088676 28696 external_mini_cluster.cc:1428] Setting key 87181cc8b4e11791cd3ec0f4aa51565b
Aug 10 00:45:28 dist-test-slave-1nl6 krb5kdc[2248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322727, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:45:28.105413 2599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:28.087084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:42197 (local address 127.28.6.62:45301)
0810 00:45:28.087620 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:28.087628 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:28.088659 (+ 1031us) server_negotiation.cc:407] Connection header received
0810 00:45:28.091040 (+ 2381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:28.091046 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:28.091130 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:28.091258 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:28.092691 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:28.093843 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:28.095429 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:28.096171 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:28.096325 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:28.099721 (+ 3396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:28.099726 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:28.099729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:28.099760 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:28.102345 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:28.103167 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:28.103172 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:28.103174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:28.103290 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:28.103735 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:28.103740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:28.103744 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:28.103999 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:28.104157 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:28.104835 (+ 678us) server_negotiation.cc:299] Negotiation successful
0810 00:45:28.105056 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":264,"threads_started":1}
I20260810 00:45:28.106416 2600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:28.087640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:45301 (local address 127.28.6.1:42197)
0810 00:45:28.088485 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:28.088533 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:28.090700 (+ 2167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:28.091455 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:28.091462 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:28.091958 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:28.092508 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:28.092525 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:28.093990 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:28.093994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:28.095279 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:28.095292 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:28.096381 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:28.096385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:28.096558 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:28.097219 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:28.097242 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:28.099494 (+ 2252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:28.102497 (+ 3003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:28.102505 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:28.102520 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:28.103033 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:28.103406 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:28.103411 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:28.103415 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:28.103602 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:28.104133 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:28.104141 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:28.104458 (+ 317us) client_negotiation.cc:769] Sending connection context
0810 00:45:28.104719 (+ 261us) client_negotiation.cc:240] Negotiation successful
0810 00:45:28.105228 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":401,"threads_started":1}
I20260810 00:45:28.108031 2595 heartbeater.cc:347] Connected to a master server at 127.28.6.62:45301
I20260810 00:45:28.108388 2595 heartbeater.cc:464] Registering TS with master...
I20260810 00:45:28.109257 2595 heartbeater.cc:511] Master 127.28.6.62:45301 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.6.2@KRBTEST.COM; defaulting to no policy
I20260810 00:45:28.111793 2311 ts_manager.cc:194] Registered new tserver with Master: 26fd688e3b6244deab1ce974911358c0 (127.28.6.1:42751)
I20260810 00:45:28.113241 2311 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.1@KRBTEST.COM'} at 127.28.6.1:42197
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab.
Entry for principal kudu/127.28.6.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab.
I20260810 00:45:28.149818 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.2:0
--local_ip_for_outbound_sockets=127.28.6.2
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:45301
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:41179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:45:28.485638 2604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:28.485886 2604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:28.486047 2604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:28.493359 2604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:28.493480 2604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:28.493682 2604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.2
I20260810 00:45:28.503163 2604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:41179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:45301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.2
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:28.504379 2604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:28.505604 2604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:28.518628 2610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:28.518802 2615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:28.518826 2612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:28.519766 2604 server_base.cc:1034] running on GCE node
I20260810 00:45:28.520339 2604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:28.521250 2604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:28.522475 2604 hybrid_clock.cc:648] HybridClock initialized: now 1786322728522431 us; error 56 us; skew 500 ppm
I20260810 00:45:28.522780 2604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:45:28 dist-test-slave-1nl6 krb5kdc[2248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322728, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:28.530117 2604 init.cc:377] Logged in from keytab as kudu/127.28.6.2@KRBTEST.COM (short username kudu)
I20260810 00:45:28.531355 2604 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:28.531489 2604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:28.531865 2604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:28.534723 2604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-1/data/instance:
uuid: "a607931bade54c0ab1e23ad3b04a8b2f"
format_stamp: "Formatted at 2026-08-10 00:45:28 on dist-test-slave-1nl6"
server_key: "6221b4c9fc1507277df0e54db8dc2d16"
server_key_iv: "af6ea881e544d79ae923c4ca4554adb7"
server_key_version: "encryptionkey@0"
I20260810 00:45:28.535651 2604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance:
uuid: "a607931bade54c0ab1e23ad3b04a8b2f"
format_stamp: "Formatted at 2026-08-10 00:45:28 on dist-test-slave-1nl6"
server_key: "6221b4c9fc1507277df0e54db8dc2d16"
server_key_iv: "af6ea881e544d79ae923c4ca4554adb7"
server_key_version: "encryptionkey@0"
I20260810 00:45:28.540676 2604 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260810 00:45:28.544078 2624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:28.545372 2604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 00:45:28.545557 2604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-1/wal
uuid: "a607931bade54c0ab1e23ad3b04a8b2f"
format_stamp: "Formatted at 2026-08-10 00:45:28 on dist-test-slave-1nl6"
server_key: "6221b4c9fc1507277df0e54db8dc2d16"
server_key_iv: "af6ea881e544d79ae923c4ca4554adb7"
server_key_version: "encryptionkey@0"
I20260810 00:45:28.545790 2604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:28.562331 2604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:28.566109 2604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:28.566385 2604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:28.567402 2604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:45:28.568745 2604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:45:28.568879 2604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:28.569016 2604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:45:28.569108 2604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:28.603926 2604 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.2:43191
I20260810 00:45:28.604010 2799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.2:43191 every 8 connection(s)
I20260810 00:45:28.604838 2604 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
I20260810 00:45:28.605377 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 2604
I20260810 00:45:28.605588 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance
I20260810 00:45:28.606204 28696 external_mini_cluster.cc:1428] Setting key 480b9ee3d63f2d0d57dacf6792f6073c
Aug 10 00:45:28 dist-test-slave-1nl6 krb5kdc[2248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322728, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:45:28.625444 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:28.607469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:40971 (local address 127.28.6.62:45301)
0810 00:45:28.607923 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:28.607929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:28.610979 (+ 3050us) server_negotiation.cc:407] Connection header received
0810 00:45:28.611135 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:28.611140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:28.611241 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:28.611364 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:28.612885 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:28.613605 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:28.615254 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:28.615919 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:28.616056 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:28.619712 (+ 3656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:28.619717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:28.619720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:28.619753 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:28.622278 (+ 2525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:28.623026 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:28.623031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:28.623037 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:28.623171 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:28.623614 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:28.623618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:28.623620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:28.623882 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:28.624043 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:28.624785 (+ 742us) server_negotiation.cc:299] Negotiation successful
0810 00:45:28.625016 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":224,"threads_started":1}
I20260810 00:45:28.626361 2804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:28.608074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:45301 (local address 127.28.6.2:40971)
0810 00:45:28.608981 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:28.609040 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:28.610802 (+ 1762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:28.611604 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:28.611612 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:28.612134 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:28.612701 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:28.612729 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:28.613783 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:28.613788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:28.615086 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:28.615099 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:28.616159 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:28.616164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:28.616354 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:28.617055 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:28.617077 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:28.619491 (+ 2414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:28.622426 (+ 2935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:28.622434 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:28.622448 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:28.622874 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:28.623290 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:28.623295 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:28.623299 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:28.623483 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:28.624024 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:28.624032 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:28.624345 (+ 313us) client_negotiation.cc:769] Sending connection context
0810 00:45:28.624666 (+ 321us) client_negotiation.cc:240] Negotiation successful
0810 00:45:28.625167 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":422,"threads_started":1}
I20260810 00:45:28.627764 2801 heartbeater.cc:347] Connected to a master server at 127.28.6.62:45301
I20260810 00:45:28.628078 2801 heartbeater.cc:464] Registering TS with master...
I20260810 00:45:28.628727 2801 heartbeater.cc:511] Master 127.28.6.62:45301 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.6.3@KRBTEST.COM; defaulting to no policy
I20260810 00:45:28.630208 2311 ts_manager.cc:194] Registered new tserver with Master: a607931bade54c0ab1e23ad3b04a8b2f (127.28.6.2:43191)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.3@KRBTEST.COM" created.
I20260810 00:45:28.631259 2311 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.2@KRBTEST.COM'} at 127.28.6.2:40971
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab.
Entry for principal kudu/127.28.6.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab.
I20260810 00:45:28.654963 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.3:0
--local_ip_for_outbound_sockets=127.28.6.3
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:45301
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:41179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:45:28.989923 2809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:28.990187 2809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:28.990288 2809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:28.997242 2809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:28.997335 2809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:28.997455 2809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.3
I20260810 00:45:29.006768 2809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:41179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:45301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.3
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:29.008028 2809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:29.009346 2809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:29.024024 2815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:29.023984 2816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:29.024060 2819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:29.025084 2809 server_base.cc:1034] running on GCE node
I20260810 00:45:29.025776 2809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:29.026840 2809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:29.028092 2809 hybrid_clock.cc:648] HybridClock initialized: now 1786322729028044 us; error 37 us; skew 500 ppm
I20260810 00:45:29.028415 2809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:45:29 dist-test-slave-1nl6 krb5kdc[2248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322729, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:29.031555 2809 init.cc:377] Logged in from keytab as kudu/127.28.6.3@KRBTEST.COM (short username kudu)
I20260810 00:45:29.032783 2809 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:29.032927 2809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:29.033274 2809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:29.036412 2809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-2/data/instance:
uuid: "24ca64d28ea04f0d9fadd163cc53b253"
format_stamp: "Formatted at 2026-08-10 00:45:29 on dist-test-slave-1nl6"
server_key: "81bc51b1af57f180ed9ca68007fbea97"
server_key_iv: "0a32a11addc7b21d31e05a163e6be4c0"
server_key_version: "encryptionkey@0"
I20260810 00:45:29.037223 2809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance:
uuid: "24ca64d28ea04f0d9fadd163cc53b253"
format_stamp: "Formatted at 2026-08-10 00:45:29 on dist-test-slave-1nl6"
server_key: "81bc51b1af57f180ed9ca68007fbea97"
server_key_iv: "0a32a11addc7b21d31e05a163e6be4c0"
server_key_version: "encryptionkey@0"
I20260810 00:45:29.043005 2809 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 00:45:29.046375 2829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:29.047719 2809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 00:45:29.047909 2809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-2/wal
uuid: "24ca64d28ea04f0d9fadd163cc53b253"
format_stamp: "Formatted at 2026-08-10 00:45:29 on dist-test-slave-1nl6"
server_key: "81bc51b1af57f180ed9ca68007fbea97"
server_key_iv: "0a32a11addc7b21d31e05a163e6be4c0"
server_key_version: "encryptionkey@0"
I20260810 00:45:29.048130 2809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:29.073046 2809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:29.076861 2809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:29.077147 2809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:29.078070 2809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:45:29.079420 2809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:45:29.079545 2809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:29.079680 2809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:45:29.079769 2809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:29.114100 2809 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.3:37003
I20260810 00:45:29.114164 2988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.3:37003 every 8 connection(s)
I20260810 00:45:29.114961 2809 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
I20260810 00:45:29.116940 2595 heartbeater.cc:503] Master 127.28.6.62:45301 was elected leader, sending a full tablet report...
I20260810 00:45:29.123396 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 2809
I20260810 00:45:29.123600 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance
I20260810 00:45:29.124104 28696 external_mini_cluster.cc:1428] Setting key ab967b9b857ddbaac7b68caa2dd1c0bd
Aug 10 00:45:29 dist-test-slave-1nl6 krb5kdc[2248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322729, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:45:29.135932 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:29.117523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:34695 (local address 127.28.6.62:45301)
0810 00:45:29.117724 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:29.117731 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:29.119011 (+ 1280us) server_negotiation.cc:407] Connection header received
0810 00:45:29.121276 (+ 2265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:29.121280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:29.121364 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:29.121451 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:29.122977 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:29.124124 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:29.126183 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:29.126780 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:29.126940 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:29.130435 (+ 3495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:29.130440 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:29.130444 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:29.130482 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:29.133140 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:29.133834 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:29.133838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:29.133841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:29.133957 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:29.134359 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:29.134362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:29.134364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:29.134615 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:29.134731 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:29.135531 (+ 800us) server_negotiation.cc:299] Negotiation successful
0810 00:45:29.135692 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 00:45:29.136895 2993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:29.117984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:45301 (local address 127.28.6.3:34695)
0810 00:45:29.118841 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:29.118879 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:29.121020 (+ 2141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:29.121685 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:29.121693 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:29.122211 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:29.122751 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:29.122769 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:29.124297 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:29.124301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:29.126007 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:29.126023 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:29.127079 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:29.127089 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:29.127444 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:29.128069 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:29.128092 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:29.130243 (+ 2151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:29.133290 (+ 3047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:29.133296 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:29.133308 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:29.133724 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:29.134053 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:29.134056 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:29.134059 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:29.134261 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:29.134731 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:29.134736 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:29.135189 (+ 453us) client_negotiation.cc:769] Sending connection context
0810 00:45:29.135436 (+ 247us) client_negotiation.cc:240] Negotiation successful
0810 00:45:29.135798 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":380,"threads_started":1}
I20260810 00:45:29.138314 2989 heartbeater.cc:347] Connected to a master server at 127.28.6.62:45301
I20260810 00:45:29.138664 2989 heartbeater.cc:464] Registering TS with master...
I20260810 00:45:29.139451 2989 heartbeater.cc:511] Master 127.28.6.62:45301 requested a full tablet report, sending...
I20260810 00:45:29.140882 2311 ts_manager.cc:194] Registered new tserver with Master: 24ca64d28ea04f0d9fadd163cc53b253 (127.28.6.3:37003)
I20260810 00:45:29.141814 2311 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.3@KRBTEST.COM'} at 127.28.6.3:34695
I20260810 00:45:29.151734 28696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 00:45:29.167724 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:29.156254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53050 (local address 127.28.6.62:45301)
0810 00:45:29.156463 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:29.156472 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:29.156935 (+ 463us) server_negotiation.cc:407] Connection header received
0810 00:45:29.157061 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:29.157065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:29.157122 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:29.157198 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:29.158541 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:29.159285 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:29.160487 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:29.160858 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:29.160949 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:29.163127 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:29.163131 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:29.163135 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:29.163160 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:29.165429 (+ 2269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:29.166009 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:29.166013 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:29.166015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:29.166132 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:29.166456 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:29.166459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:29.166461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:29.166729 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:29.166989 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:29.167235 (+ 246us) server_negotiation.cc:299] Negotiation successful
0810 00:45:29.167381 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 00:45:29.171501 2311 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 00:45:29.178176 2311 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 00:45:29.202049 3008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:29.190170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:43191 (local address 127.0.0.1:40946)
0810 00:45:29.191516 (+ 1346us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:29.191553 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:29.191725 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:29.194094 (+ 2369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:29.194101 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:29.194144 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:29.194550 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:29.194564 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:29.196738 (+ 2174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:29.196744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:29.199439 (+ 2695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:29.199458 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:29.201047 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:29.201053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:29.201267 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:29.201323 (+ 56us) client_negotiation.cc:769] Sending connection context
0810 00:45:29.201432 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 00:45:29.201631 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1104,"mutex_wait_us":1,"thread_start_us":1906,"threads_started":1}
I20260810 00:45:29.202557 3004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:29.190512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:42751 (local address 127.0.0.1:57020)
0810 00:45:29.192404 (+ 1892us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:29.192426 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:29.192561 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:29.194925 (+ 2364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:29.194933 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:29.194968 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:29.195440 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:29.195452 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:29.197218 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:29.197224 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:29.199641 (+ 2417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:29.199651 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:29.201615 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:29.201622 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:29.201819 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:29.201852 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 00:45:29.201929 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 00:45:29.202077 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1731,"mutex_wait_us":22,"thread_start_us":693,"threads_started":1}
I20260810 00:45:29.202618 3005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:29.190282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40946 (local address 127.28.6.2:43191)
0810 00:45:29.193444 (+ 3162us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:29.193454 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:29.193483 (+ 29us) server_negotiation.cc:407] Connection header received
0810 00:45:29.193595 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:29.193603 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:29.193844 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:29.194752 (+ 908us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:29.195022 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:29.196545 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:29.199617 (+ 3072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:29.200812 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:29.201786 (+ 974us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:29.201975 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:29.202063 (+ 88us) server_negotiation.cc:299] Negotiation successful
0810 00:45:29.202249 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2988,"thread_start_us":2892,"threads_started":1}
I20260810 00:45:29.203409 3009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:29.191048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57020 (local address 127.28.6.1:42751)
0810 00:45:29.194324 (+ 3276us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:29.194332 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:29.194361 (+ 29us) server_negotiation.cc:407] Connection header received
0810 00:45:29.194460 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:29.194468 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:29.194699 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:29.194926 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:29.195635 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:29.197012 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:29.199782 (+ 2770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:29.201085 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:29.201221 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:29.201461 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:29.202991 (+ 1530us) server_negotiation.cc:299] Negotiation successful
0810 00:45:29.203186 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3092,"thread_start_us":1797,"threads_started":1}
I20260810 00:45:29.203430 3012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:29.193185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53384 (local address 127.28.6.3:37003)
0810 00:45:29.195629 (+ 2444us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:29.195637 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:29.195666 (+ 29us) server_negotiation.cc:407] Connection header received
0810 00:45:29.195753 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:29.195759 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:29.195961 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:29.196148 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:29.196754 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:29.198071 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:29.200211 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:29.201646 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:29.201771 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:29.202015 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:29.202947 (+ 932us) server_negotiation.cc:299] Negotiation successful
0810 00:45:29.203114 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2278,"thread_start_us":2123,"threads_started":1}
I20260810 00:45:29.203101 3010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:29.192568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:37003 (local address 127.0.0.1:53384)
0810 00:45:29.193798 (+ 1230us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:29.193820 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:29.193960 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:29.196169 (+ 2209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:29.196174 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:29.196200 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:29.196530 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:29.196542 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:29.198217 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:29.198224 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:29.200016 (+ 1792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:29.200046 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:29.202159 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:29.202167 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:29.202339 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:29.202384 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 00:45:29.202548 (+ 164us) client_negotiation.cc:240] Negotiation successful
0810 00:45:29.202705 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1083,"mutex_wait_us":25,"thread_start_us":938,"threads_started":1}
I20260810 00:45:29.206612 2694 tablet_service.cc:1467] Processing CreateTablet for tablet cbcb5c635f01463eb463647fdd7fcdba (DEFAULT_TABLE table=test-table [id=f467f98128484cc1afc090f86dbc95ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:29.207260 2895 tablet_service.cc:1467] Processing CreateTablet for tablet cbcb5c635f01463eb463647fdd7fcdba (DEFAULT_TABLE table=test-table [id=f467f98128484cc1afc090f86dbc95ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:29.207204 2503 tablet_service.cc:1467] Processing CreateTablet for tablet cbcb5c635f01463eb463647fdd7fcdba (DEFAULT_TABLE table=test-table [id=f467f98128484cc1afc090f86dbc95ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:29.207978 2694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbcb5c635f01463eb463647fdd7fcdba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:29.208536 2503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbcb5c635f01463eb463647fdd7fcdba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:29.208555 2895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbcb5c635f01463eb463647fdd7fcdba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:29.220650 3013 tablet_bootstrap.cc:492] T cbcb5c635f01463eb463647fdd7fcdba P a607931bade54c0ab1e23ad3b04a8b2f: Bootstrap starting.
I20260810 00:45:29.223268 3014 tablet_bootstrap.cc:492] T cbcb5c635f01463eb463647fdd7fcdba P 26fd688e3b6244deab1ce974911358c0: Bootstrap starting.
I20260810 00:45:29.224519 3013 tablet_bootstrap.cc:654] T cbcb5c635f01463eb463647fdd7fcdba P a607931bade54c0ab1e23ad3b04a8b2f: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:29.224917 3015 tablet_bootstrap.cc:492] T cbcb5c635f01463eb463647fdd7fcdba P 24ca64d28ea04f0d9fadd163cc53b253: Bootstrap starting.
I20260810 00:45:29.225867 3013 log.cc:824] T cbcb5c635f01463eb463647fdd7fcdba P a607931bade54c0ab1e23ad3b04a8b2f: Log is configured to *not* fsync() on all Append() calls
I20260810 00:45:29.227243 3014 tablet_bootstrap.cc:654] T cbcb5c635f01463eb463647fdd7fcdba P 26fd688e3b6244deab1ce974911358c0: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:29.228449 3014 log.cc:824] T cbcb5c635f01463eb463647fdd7fcdba P 26fd688e3b6244deab1ce974911358c0: Log is configured to *not* fsync() on all Append() calls
I20260810 00:45:29.228507 3015 tablet_bootstrap.cc:654] T cbcb5c635f01463eb463647fdd7fcdba P 24ca64d28ea04f0d9fadd163cc53b253: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:29.229229 3013 tablet_bootstrap.cc:492] T cbcb5c635f01463eb463647fdd7fcdba P a607931bade54c0ab1e23ad3b04a8b2f: No bootstrap required, opened a new log
I20260810 00:45:29.229496 3013 ts_tablet_manager.cc:1397] T cbcb5c635f01463eb463647fdd7fcdba P a607931bade54c0ab1e23ad3b04a8b2f: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260810 00:45:29.229679 3015 log.cc:824] T cbcb5c635f01463eb463647fdd7fcdba P 24ca64d28ea04f0d9fadd163cc53b253: Log is configured to *not* fsync() on all Append() calls
I20260810 00:45:29.231583 3014 tablet_bootstrap.cc:492] T cbcb5c635f01463eb463647fdd7fcdba P 26fd688e3b6244deab1ce974911358c0: No bootstrap required, opened a new log
I20260810 00:45:29.231856 3014 ts_tablet_manager.cc:1397] T cbcb5c635f01463eb463647fdd7fcdba P 26fd688e3b6244deab1ce974911358c0: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260810 00:45:29.232585 3015 tablet_bootstrap.cc:492] T cbcb5c635f01463eb463647fdd7fcdba P 24ca64d28ea04f0d9fadd163cc53b253: No bootstrap required, opened a new log
I20260810 00:45:29.232837 3015 ts_tablet_manager.cc:1397] T cbcb5c635f01463eb463647fdd7fcdba P 24ca64d28ea04f0d9fadd163cc53b253: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 00:45:29.240254 3013 raft_consensus.cc:348] T cbcb5c635f01463eb463647fdd7fcdba P a607931bade54c0ab1e23ad3b04a8b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a607931bade54c0ab1e23ad3b04a8b2f" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 43191 } } peers { permanent_uuid: "24ca64d28ea04f0d9fadd163cc53b253" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 37003 } } peers { permanent_uuid: "26fd688e3b6244deab1ce974911358c0" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 42751 } }
I20260810 00:45:29.240700 3013 raft_consensus.cc:374] T cbcb5c635f01463eb463647fdd7fcdba P a607931bade54c0ab1e23ad3b04a8b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:29.240806 3013 raft_consensus.cc:729] T cbcb5c635f01463eb463647fdd7fcdba P a607931bade54c0ab1e23ad3b04a8b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a607931bade54c0ab1e23ad3b04a8b2f, State: Initialized, Role: FOLLOWER
I20260810 00:45:29.240859 3015 raft_consensus.cc:348] T cbcb5c635f01463eb463647fdd7fcdba P 24ca64d28ea04f0d9fadd163cc53b253 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a607931bade54c0ab1e23ad3b04a8b2f" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 43191 } } peers { permanent_uuid: "24ca64d28ea04f0d9fadd163cc53b253" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 37003 } } peers { permanent_uuid: "26fd688e3b6244deab1ce974911358c0" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 42751 } }
I20260810 00:45:29.241283 3015 raft_consensus.cc:374] T cbcb5c635f01463eb463647fdd7fcdba P 24ca64d28ea04f0d9fadd163cc53b253 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:29.241415 3015 raft_consensus.cc:729] T cbcb5c635f01463eb463647fdd7fcdba P 24ca64d28ea04f0d9fadd163cc53b253 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24ca64d28ea04f0d9fadd163cc53b253, State: Initialized, Role: FOLLOWER
I20260810 00:45:29.241427 3013 consensus_queue.cc:260] T cbcb5c635f01463eb463647fdd7fcdba P a607931bade54c0ab1e23ad3b04a8b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a607931bade54c0ab1e23ad3b04a8b2f" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 43191 } } peers { permanent_uuid: "24ca64d28ea04f0d9fadd163cc53b253" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 37003 } } peers { permanent_uuid: "26fd688e3b6244deab1ce974911358c0" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 42751 } }
I20260810 00:45:29.242017 3015 consensus_queue.cc:260] T cbcb5c635f01463eb463647fdd7fcdba P 24ca64d28ea04f0d9fadd163cc53b253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a607931bade54c0ab1e23ad3b04a8b2f" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 43191 } } peers { permanent_uuid: "24ca64d28ea04f0d9fadd163cc53b253" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 37003 } } peers { permanent_uuid: "26fd688e3b6244deab1ce974911358c0" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 42751 } }
I20260810 00:45:29.242278 3014 raft_consensus.cc:348] T cbcb5c635f01463eb463647fdd7fcdba P 26fd688e3b6244deab1ce974911358c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a607931bade54c0ab1e23ad3b04a8b2f" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 43191 } } peers { permanent_uuid: "24ca64d28ea04f0d9fadd163cc53b253" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 37003 } } peers { permanent_uuid: "26fd688e3b6244deab1ce974911358c0" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 42751 } }
I20260810 00:45:29.242661 3014 raft_consensus.cc:374] T cbcb5c635f01463eb463647fdd7fcdba P 26fd688e3b6244deab1ce974911358c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:29.242690 2801 heartbeater.cc:503] Master 127.28.6.62:45301 was elected leader, sending a full tablet report...
I20260810 00:45:29.242770 3014 raft_consensus.cc:729] T cbcb5c635f01463eb463647fdd7fcdba P 26fd688e3b6244deab1ce974911358c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26fd688e3b6244deab1ce974911358c0, State: Initialized, Role: FOLLOWER
I20260810 00:45:29.243381 3013 ts_tablet_manager.cc:1428] T cbcb5c635f01463eb463647fdd7fcdba P a607931bade54c0ab1e23ad3b04a8b2f: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260810 00:45:29.243382 3014 consensus_queue.cc:260] T cbcb5c635f01463eb463647fdd7fcdba P 26fd688e3b6244deab1ce974911358c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a607931bade54c0ab1e23ad3b04a8b2f" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 43191 } } peers { permanent_uuid: "24ca64d28ea04f0d9fadd163cc53b253" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 37003 } } peers { permanent_uuid: "26fd688e3b6244deab1ce974911358c0" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 42751 } }
I20260810 00:45:29.244884 2989 heartbeater.cc:503] Master 127.28.6.62:45301 was elected leader, sending a full tablet report...
I20260810 00:45:29.245103 3014 ts_tablet_manager.cc:1428] T cbcb5c635f01463eb463647fdd7fcdba P 26fd688e3b6244deab1ce974911358c0: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260810 00:45:29.247176 3015 ts_tablet_manager.cc:1428] T cbcb5c635f01463eb463647fdd7fcdba P 24ca64d28ea04f0d9fadd163cc53b253: Time spent starting tablet: real 0.014s user 0.008s sys 0.006s
W20260810 00:45:29.337064 2596 tablet.cc:2369] T cbcb5c635f01463eb463647fdd7fcdba P 26fd688e3b6244deab1ce974911358c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 00:45:29.356869 2802 tablet.cc:2369] T cbcb5c635f01463eb463647fdd7fcdba P a607931bade54c0ab1e23ad3b04a8b2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 00:45:29.366456 2990 tablet.cc:2369] T cbcb5c635f01463eb463647fdd7fcdba P 24ca64d28ea04f0d9fadd163cc53b253: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:45:29.451346 3026 raft_consensus.cc:482] T cbcb5c635f01463eb463647fdd7fcdba P 26fd688e3b6244deab1ce974911358c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:45:29.451694 3026 raft_consensus.cc:504] T cbcb5c635f01463eb463647fdd7fcdba P 26fd688e3b6244deab1ce974911358c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a607931bade54c0ab1e23ad3b04a8b2f" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 43191 } } peers { permanent_uuid: "24ca64d28ea04f0d9fadd163cc53b253" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 37003 } } peers { permanent_uuid: "26fd688e3b6244deab1ce974911358c0" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 42751 } }
I20260810 00:45:29.453577 3026 leader_election.cc:288] T cbcb5c635f01463eb463647fdd7fcdba P 26fd688e3b6244deab1ce974911358c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a607931bade54c0ab1e23ad3b04a8b2f (127.28.6.2:43191), 24ca64d28ea04f0d9fadd163cc53b253 (127.28.6.3:37003)
I20260810 00:45:29.462244 3028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:29.453959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:43191 (local address 127.28.6.1:36805)
0810 00:45:29.454762 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:29.454784 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:29.454966 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:29.455588 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:29.455593 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:29.455639 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:29.456070 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:29.456084 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:29.457648 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:29.457654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:29.460116 (+ 2462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:29.460132 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:29.461569 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:29.461573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:29.461685 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:29.461710 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 00:45:29.461787 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 00:45:29.461871 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":477,"threads_started":1}
I20260810 00:45:29.462265 3005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:29.454027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:36805 (local address 127.28.6.2:43191)
0810 00:45:29.454242 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:29.454247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:29.454886 (+ 639us) server_negotiation.cc:407] Connection header received
0810 00:45:29.455149 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:29.455156 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:29.455241 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:29.455360 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:29.456301 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:29.457498 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:29.460283 (+ 2785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:29.461368 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:29.461518 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:29.461618 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:29.461849 (+ 231us) server_negotiation.cc:299] Negotiation successful
0810 00:45:29.462003 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 00:45:29.462873 3012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:29.455206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:45013 (local address 127.28.6.3:37003)
0810 00:45:29.455494 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:29.455501 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:29.455919 (+ 418us) server_negotiation.cc:407] Connection header received
0810 00:45:29.456283 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:29.456288 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:29.456371 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:29.456437 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:29.457274 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:29.458389 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:29.460676 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:29.461914 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:29.462042 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:29.462142 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:29.462497 (+ 355us) server_negotiation.cc:299] Negotiation successful
0810 00:45:29.462629 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260810 00:45:29.462926 3029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:29.454954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:37003 (local address 127.28.6.1:45013)
0810 00:45:29.455811 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:29.455832 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:29.456086 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:29.456632 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:29.456638 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:29.456665 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:29.457045 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:29.457057 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:29.458549 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:29.458556 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:29.460530 (+ 1974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:29.460542 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:29.462137 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:29.462142 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:29.462339 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:29.462371 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 00:45:29.462463 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 00:45:29.462582 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":1845,"threads_started":1}
I20260810 00:45:29.463424 2724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbcb5c635f01463eb463647fdd7fcdba" candidate_uuid: "26fd688e3b6244deab1ce974911358c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a607931bade54c0ab1e23ad3b04a8b2f" is_pre_election: true
I20260810 00:45:29.463867 2724 raft_consensus.cc:2454] T cbcb5c635f01463eb463647fdd7fcdba P a607931bade54c0ab1e23ad3b04a8b2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26fd688e3b6244deab1ce974911358c0 in term 0.
I20260810 00:45:29.463811 2925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbcb5c635f01463eb463647fdd7fcdba" candidate_uuid: "26fd688e3b6244deab1ce974911358c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ca64d28ea04f0d9fadd163cc53b253" is_pre_election: true
I20260810 00:45:29.464195 2925 raft_consensus.cc:2454] T cbcb5c635f01463eb463647fdd7fcdba P 24ca64d28ea04f0d9fadd163cc53b253 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26fd688e3b6244deab1ce974911358c0 in term 0.
I20260810 00:45:29.464540 2438 leader_election.cc:302] T cbcb5c635f01463eb463647fdd7fcdba P 26fd688e3b6244deab1ce974911358c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26fd688e3b6244deab1ce974911358c0, a607931bade54c0ab1e23ad3b04a8b2f; no voters:
I20260810 00:45:29.464936 3026 raft_consensus.cc:2781] T cbcb5c635f01463eb463647fdd7fcdba P 26fd688e3b6244deab1ce974911358c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 00:45:29.465087 3026 raft_consensus.cc:482] T cbcb5c635f01463eb463647fdd7fcdba P 26fd688e3b6244deab1ce974911358c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 00:45:29.465248 3026 raft_consensus.cc:3037] T cbcb5c635f01463eb463647fdd7fcdba P 26fd688e3b6244deab1ce974911358c0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:29.467478 3026 raft_consensus.cc:504] T cbcb5c635f01463eb463647fdd7fcdba P 26fd688e3b6244deab1ce974911358c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a607931bade54c0ab1e23ad3b04a8b2f" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 43191 } } peers { permanent_uuid: "24ca64d28ea04f0d9fadd163cc53b253" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 37003 } } peers { permanent_uuid: "26fd688e3b6244deab1ce974911358c0" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 42751 } }
I20260810 00:45:29.468034 3026 leader_election.cc:288] T cbcb5c635f01463eb463647fdd7fcdba P 26fd688e3b6244deab1ce974911358c0 [CANDIDATE]: Term 1 election: Requested vote from peers a607931bade54c0ab1e23ad3b04a8b2f (127.28.6.2:43191), 24ca64d28ea04f0d9fadd163cc53b253 (127.28.6.3:37003)
I20260810 00:45:29.468477 2724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbcb5c635f01463eb463647fdd7fcdba" candidate_uuid: "26fd688e3b6244deab1ce974911358c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a607931bade54c0ab1e23ad3b04a8b2f"
I20260810 00:45:29.468568 2925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbcb5c635f01463eb463647fdd7fcdba" candidate_uuid: "26fd688e3b6244deab1ce974911358c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ca64d28ea04f0d9fadd163cc53b253"
I20260810 00:45:29.468667 2724 raft_consensus.cc:3037] T cbcb5c635f01463eb463647fdd7fcdba P a607931bade54c0ab1e23ad3b04a8b2f [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:29.468817 2925 raft_consensus.cc:3037] T cbcb5c635f01463eb463647fdd7fcdba P 24ca64d28ea04f0d9fadd163cc53b253 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:29.470884 2724 raft_consensus.cc:2454] T cbcb5c635f01463eb463647fdd7fcdba P a607931bade54c0ab1e23ad3b04a8b2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26fd688e3b6244deab1ce974911358c0 in term 1.
I20260810 00:45:29.470990 2925 raft_consensus.cc:2454] T cbcb5c635f01463eb463647fdd7fcdba P 24ca64d28ea04f0d9fadd163cc53b253 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26fd688e3b6244deab1ce974911358c0 in term 1.
I20260810 00:45:29.471304 2438 leader_election.cc:302] T cbcb5c635f01463eb463647fdd7fcdba P 26fd688e3b6244deab1ce974911358c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26fd688e3b6244deab1ce974911358c0, a607931bade54c0ab1e23ad3b04a8b2f; no voters:
I20260810 00:45:29.471601 3026 raft_consensus.cc:2781] T cbcb5c635f01463eb463647fdd7fcdba P 26fd688e3b6244deab1ce974911358c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:45:29.471998 3026 raft_consensus.cc:686] T cbcb5c635f01463eb463647fdd7fcdba P 26fd688e3b6244deab1ce974911358c0 [term 1 LEADER]: Becoming Leader. State: Replica: 26fd688e3b6244deab1ce974911358c0, State: Running, Role: LEADER
I20260810 00:45:29.472577 3026 consensus_queue.cc:237] T cbcb5c635f01463eb463647fdd7fcdba P 26fd688e3b6244deab1ce974911358c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a607931bade54c0ab1e23ad3b04a8b2f" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 43191 } } peers { permanent_uuid: "24ca64d28ea04f0d9fadd163cc53b253" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 37003 } } peers { permanent_uuid: "26fd688e3b6244deab1ce974911358c0" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 42751 } }
I20260810 00:45:29.477036 2311 catalog_manager.cc:5495] T cbcb5c635f01463eb463647fdd7fcdba P 26fd688e3b6244deab1ce974911358c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26fd688e3b6244deab1ce974911358c0 (127.28.6.1). New cstate: current_term: 1 leader_uuid: "26fd688e3b6244deab1ce974911358c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a607931bade54c0ab1e23ad3b04a8b2f" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 43191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24ca64d28ea04f0d9fadd163cc53b253" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 37003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26fd688e3b6244deab1ce974911358c0" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 42751 } health_report { overall_health: HEALTHY } } }
I20260810 00:45:29.494222 3009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:29.488233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57036 (local address 127.28.6.1:42751)
0810 00:45:29.488445 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:29.488449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:29.488471 (+ 22us) server_negotiation.cc:407] Connection header received
0810 00:45:29.488680 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:29.488684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:29.488768 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:29.488892 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:45:29.489306 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:29.489985 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:29.492004 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:29.492436 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:29.492539 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:29.493095 (+ 556us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:45:29.493600 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:45:29.493712 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:29.493854 (+ 142us) server_negotiation.cc:299] Negotiation successful
0810 00:45:29.493923 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 00:45:29.503086 2925 raft_consensus.cc:1261] T cbcb5c635f01463eb463647fdd7fcdba P 24ca64d28ea04f0d9fadd163cc53b253 [term 1 FOLLOWER]: Refusing update from remote peer 26fd688e3b6244deab1ce974911358c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:45:29.503114 2724 raft_consensus.cc:1261] T cbcb5c635f01463eb463647fdd7fcdba P a607931bade54c0ab1e23ad3b04a8b2f [term 1 FOLLOWER]: Refusing update from remote peer 26fd688e3b6244deab1ce974911358c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:45:29.504065 3030 consensus_queue.cc:1035] T cbcb5c635f01463eb463647fdd7fcdba P 26fd688e3b6244deab1ce974911358c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ca64d28ea04f0d9fadd163cc53b253" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 37003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:45:29.504436 3026 consensus_queue.cc:1035] T cbcb5c635f01463eb463647fdd7fcdba P 26fd688e3b6244deab1ce974911358c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a607931bade54c0ab1e23ad3b04a8b2f" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 43191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:45:29.515676 3037 mvcc.cc:205] Tried to move back new op lower bound from 7316777900034838528 to 7316777899923427328. Current Snapshot: MvccSnapshot[applied={T|T < 7316777900034838528}]
I20260810 00:45:29.518312 3040 mvcc.cc:205] Tried to move back new op lower bound from 7316777900034838528 to 7316777899923427328. Current Snapshot: MvccSnapshot[applied={T|T < 7316777900034838528}]
I20260810 00:45:29.529031 2311 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:45:29.529309 2311 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:45:29.532703 2311 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 79155b0a285b4a9b86dbe7094f3f0115: Sending DeleteTablet for 3 replicas of tablet cbcb5c635f01463eb463647fdd7fcdba
I20260810 00:45:29.533795 2895 tablet_service.cc:1514] Processing DeleteTablet for tablet cbcb5c635f01463eb463647fdd7fcdba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:45:29 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:53384
I20260810 00:45:29.533761 2694 tablet_service.cc:1514] Processing DeleteTablet for tablet cbcb5c635f01463eb463647fdd7fcdba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:45:29 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:40946
I20260810 00:45:29.534507 2503 tablet_service.cc:1514] Processing DeleteTablet for tablet cbcb5c635f01463eb463647fdd7fcdba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:45:29 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:57020
I20260810 00:45:29.535187 3053 tablet_replica.cc:330] T cbcb5c635f01463eb463647fdd7fcdba P a607931bade54c0ab1e23ad3b04a8b2f: stopping tablet replica
I20260810 00:45:29.535354 3052 tablet_replica.cc:330] T cbcb5c635f01463eb463647fdd7fcdba P 24ca64d28ea04f0d9fadd163cc53b253: stopping tablet replica
I20260810 00:45:29.535831 3052 raft_consensus.cc:2229] T cbcb5c635f01463eb463647fdd7fcdba P 24ca64d28ea04f0d9fadd163cc53b253 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:45:29.536020 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 2411
I20260810 00:45:29.536410 3053 raft_consensus.cc:2229] T cbcb5c635f01463eb463647fdd7fcdba P a607931bade54c0ab1e23ad3b04a8b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:45:29.536962 3053 raft_consensus.cc:2258] T cbcb5c635f01463eb463647fdd7fcdba P a607931bade54c0ab1e23ad3b04a8b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:45:29.537474 3052 raft_consensus.cc:2258] T cbcb5c635f01463eb463647fdd7fcdba P 24ca64d28ea04f0d9fadd163cc53b253 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:45:29.540040 3052 ts_tablet_manager.cc:1905] T cbcb5c635f01463eb463647fdd7fcdba P 24ca64d28ea04f0d9fadd163cc53b253: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:45:29.547051 3054 tablet_replica.cc:330] T cbcb5c635f01463eb463647fdd7fcdba P 26fd688e3b6244deab1ce974911358c0: stopping tablet replica
I20260810 00:45:29.547461 3054 raft_consensus.cc:2229] T cbcb5c635f01463eb463647fdd7fcdba P 26fd688e3b6244deab1ce974911358c0 [term 1 LEADER]: Raft consensus shutting down.
I20260810 00:45:29.548018 3053 ts_tablet_manager.cc:1905] T cbcb5c635f01463eb463647fdd7fcdba P a607931bade54c0ab1e23ad3b04a8b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:45:29.548203 3054 raft_consensus.cc:2258] T cbcb5c635f01463eb463647fdd7fcdba P 26fd688e3b6244deab1ce974911358c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:45:29.549026 3052 ts_tablet_manager.cc:1918] T cbcb5c635f01463eb463647fdd7fcdba P 24ca64d28ea04f0d9fadd163cc53b253: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:45:29.549270 3052 log.cc:1196] T cbcb5c635f01463eb463647fdd7fcdba P 24ca64d28ea04f0d9fadd163cc53b253: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-2/wal/wals/cbcb5c635f01463eb463647fdd7fcdba
I20260810 00:45:29.549867 3052 ts_tablet_manager.cc:1939] T cbcb5c635f01463eb463647fdd7fcdba P 24ca64d28ea04f0d9fadd163cc53b253: Deleting consensus metadata
I20260810 00:45:29.550222 3054 ts_tablet_manager.cc:1905] T cbcb5c635f01463eb463647fdd7fcdba P 26fd688e3b6244deab1ce974911358c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:45:29.551774 2289 catalog_manager.cc:4841] TS 24ca64d28ea04f0d9fadd163cc53b253 (127.28.6.3:37003): tablet cbcb5c635f01463eb463647fdd7fcdba (table test-table [id=f467f98128484cc1afc090f86dbc95ee]) successfully deleted
I20260810 00:45:29.555787 3053 ts_tablet_manager.cc:1918] T cbcb5c635f01463eb463647fdd7fcdba P a607931bade54c0ab1e23ad3b04a8b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:45:29.555974 3053 log.cc:1196] T cbcb5c635f01463eb463647fdd7fcdba P a607931bade54c0ab1e23ad3b04a8b2f: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-1/wal/wals/cbcb5c635f01463eb463647fdd7fcdba
I20260810 00:45:29.556511 3053 ts_tablet_manager.cc:1939] T cbcb5c635f01463eb463647fdd7fcdba P a607931bade54c0ab1e23ad3b04a8b2f: Deleting consensus metadata
I20260810 00:45:29.557673 3054 ts_tablet_manager.cc:1918] T cbcb5c635f01463eb463647fdd7fcdba P 26fd688e3b6244deab1ce974911358c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:45:29.557855 3054 log.cc:1196] T cbcb5c635f01463eb463647fdd7fcdba P 26fd688e3b6244deab1ce974911358c0: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786322687226232-28696-0/minicluster-data/ts-0/wal/wals/cbcb5c635f01463eb463647fdd7fcdba
I20260810 00:45:29.558269 2289 catalog_manager.cc:4841] TS a607931bade54c0ab1e23ad3b04a8b2f (127.28.6.2:43191): tablet cbcb5c635f01463eb463647fdd7fcdba (table test-table [id=f467f98128484cc1afc090f86dbc95ee]) successfully deleted
I20260810 00:45:29.558421 3054 ts_tablet_manager.cc:1939] T cbcb5c635f01463eb463647fdd7fcdba P 26fd688e3b6244deab1ce974911358c0: Deleting consensus metadata
I20260810 00:45:29.560082 2287 catalog_manager.cc:4841] TS 26fd688e3b6244deab1ce974911358c0 (127.28.6.1:42751): tablet cbcb5c635f01463eb463647fdd7fcdba (table test-table [id=f467f98128484cc1afc090f86dbc95ee]) successfully deleted
Aug 10 00:45:29 dist-test-slave-1nl6 krb5kdc[2248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.1@KRBTEST.COM
I20260810 00:45:29.568813 3009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:29.537207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57042 (local address 127.28.6.1:42751)
0810 00:45:29.537377 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:29.537385 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:29.549209 (+ 11824us) server_negotiation.cc:407] Connection header received
0810 00:45:29.549571 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:29.549577 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:29.549666 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:29.549744 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:29.552878 (+ 3134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:29.554044 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:29.556025 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:29.556817 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:29.556926 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:29.561975 (+ 5049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:29.561987 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:29.562006 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:29.562049 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:29.565989 (+ 3940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:29.566647 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:29.566657 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:29.566661 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:29.566856 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:29.567243 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:29.567253 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:29.567257 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:29.567706 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:29.568006 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:29.568183 (+ 177us) server_negotiation.cc:299] Negotiation successful
0810 00:45:29.568428 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 00:45:29.569463 2588 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:45:29.680755 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 2411
I20260810 00:45:29.707283 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 2604
Aug 10 00:45:29 dist-test-slave-1nl6 krb5kdc[2248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.2@KRBTEST.COM
I20260810 00:45:29.722191 3005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:29.708104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40962 (local address 127.28.6.2:43191)
0810 00:45:29.708329 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:29.708334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:29.708353 (+ 19us) server_negotiation.cc:407] Connection header received
0810 00:45:29.708520 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:29.708524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:29.708609 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:29.708724 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:29.709895 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:29.710629 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:29.711883 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:29.712295 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:29.712396 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:29.716628 (+ 4232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:29.716642 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:29.716660 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:29.716710 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:29.719701 (+ 2991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:29.720298 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:29.720305 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:29.720308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:29.720439 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:29.720778 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:29.720781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:29.720783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:29.721145 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:29.721421 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:29.721620 (+ 199us) server_negotiation.cc:299] Negotiation successful
0810 00:45:29.721870 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 00:45:29.722893 2793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:45:29.812634 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 2604
I20260810 00:45:29.842258 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 2809
Aug 10 00:45:29 dist-test-slave-1nl6 krb5kdc[2248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.3@KRBTEST.COM
I20260810 00:45:29.857239 3012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:29.843196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53396 (local address 127.28.6.3:37003)
0810 00:45:29.843402 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:29.843408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:29.843429 (+ 21us) server_negotiation.cc:407] Connection header received
0810 00:45:29.843520 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:29.843524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:29.843591 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:29.843676 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:29.844937 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:29.845671 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:29.846881 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:29.847435 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:29.847535 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:29.851703 (+ 4168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:29.851720 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:29.851738 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:29.851773 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:29.854621 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:29.855346 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:29.855355 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:29.855357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:29.855507 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:29.855843 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:29.855846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:29.855848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:29.856207 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:29.856444 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:29.856654 (+ 210us) server_negotiation.cc:299] Negotiation successful
0810 00:45:29.856829 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 00:45:29.857967 2983 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:45:29.946967 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 2809
I20260810 00:45:29.972694 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 2265
I20260810 00:45:29.973883 2361 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:45:30.082841 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 2265
2026-08-10T00:45:30Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4524 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-10T00:45:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:45:30Z Disabled control of system clock
I20260810 00:45:30.163482 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:33549
--webserver_interface=127.28.6.62
--webserver_port=0
--builtin_ntp_servers=127.28.6.20:42745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:33549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 00:45:30.490289 3071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:30.490543 3071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:30.490625 3071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:30.497574 3071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:45:30.497669 3071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:30.497715 3071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:30.497769 3071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:45:30.497813 3071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:45:30.506572 3071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:42745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:33549
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:33549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:30.507761 3071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:30.509413 3071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:30.520792 3079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:30.520920 3084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:30.520958 3081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:30.521766 3071 server_base.cc:1034] running on GCE node
I20260810 00:45:30.522482 3071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:30.523835 3071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:30.525031 3071 hybrid_clock.cc:648] HybridClock initialized: now 1786322730524979 us; error 52 us; skew 500 ppm
I20260810 00:45:30.525321 3071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 00:45:30.527555 3071 webserver.cc:460] Webserver started at http://127.28.6.62:35407/ using document root <none> and password file <none>
I20260810 00:45:30.528149 3071 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:30.528244 3071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:30.528510 3071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:30.531275 3071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "ae090afa015d4aceb8e0d3db8a66e79e"
format_stamp: "Formatted at 2026-08-10 00:45:30 on dist-test-slave-1nl6"
server_key: "6ddce3679fdbc2783abbf8cd17b1effe"
server_key_iv: "943afe16ab44f4a04d8fd21f6fb94d05"
server_key_version: "encryptionkey@0"
I20260810 00:45:30.531988 3071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "ae090afa015d4aceb8e0d3db8a66e79e"
format_stamp: "Formatted at 2026-08-10 00:45:30 on dist-test-slave-1nl6"
server_key: "6ddce3679fdbc2783abbf8cd17b1effe"
server_key_iv: "943afe16ab44f4a04d8fd21f6fb94d05"
server_key_version: "encryptionkey@0"
I20260810 00:45:30.536685 3071 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260810 00:45:30.539897 3094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:30.541127 3071 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 00:45:30.541272 3071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "ae090afa015d4aceb8e0d3db8a66e79e"
format_stamp: "Formatted at 2026-08-10 00:45:30 on dist-test-slave-1nl6"
server_key: "6ddce3679fdbc2783abbf8cd17b1effe"
server_key_iv: "943afe16ab44f4a04d8fd21f6fb94d05"
server_key_version: "encryptionkey@0"
I20260810 00:45:30.541441 3071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:30.547070 3071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:30.547745 3071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:30.548003 3071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:30.566481 3071 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:33549
I20260810 00:45:30.566524 3167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:33549 every 8 connection(s)
I20260810 00:45:30.567812 3071 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:45:30.570755 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 3071
I20260810 00:45:30.570963 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:45:30.571544 3168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:30.571633 28696 external_mini_cluster.cc:1428] Setting key 47f6c94db5f1e8521091d2e73d9bc5d4
I20260810 00:45:30.579878 3172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:30.573637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33812 (local address 127.28.6.62:33549)
0810 00:45:30.574636 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:30.574646 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:30.574687 (+ 41us) server_negotiation.cc:407] Connection header received
0810 00:45:30.576345 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:30.576394 (+ 49us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:30.576942 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:30.577267 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:30.578372 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:30.578378 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:30.578394 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:45:30.578411 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:30.578469 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:45:30.578483 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:30.578655 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:30.578769 (+ 114us) server_negotiation.cc:299] Negotiation successful
0810 00:45:30.579048 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":385,"threads_started":1}
I20260810 00:45:30.584281 3168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae090afa015d4aceb8e0d3db8a66e79e: Bootstrap starting.
I20260810 00:45:30.587111 3168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae090afa015d4aceb8e0d3db8a66e79e: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:30.588096 3168 log.cc:824] T 00000000000000000000000000000000 P ae090afa015d4aceb8e0d3db8a66e79e: Log is configured to *not* fsync() on all Append() calls
I20260810 00:45:30.590648 3168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae090afa015d4aceb8e0d3db8a66e79e: No bootstrap required, opened a new log
I20260810 00:45:30.597910 3168 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae090afa015d4aceb8e0d3db8a66e79e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae090afa015d4aceb8e0d3db8a66e79e" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 33549 } }
I20260810 00:45:30.598269 3168 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae090afa015d4aceb8e0d3db8a66e79e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:30.598368 3168 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae090afa015d4aceb8e0d3db8a66e79e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae090afa015d4aceb8e0d3db8a66e79e, State: Initialized, Role: FOLLOWER
I20260810 00:45:30.599067 3168 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae090afa015d4aceb8e0d3db8a66e79e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae090afa015d4aceb8e0d3db8a66e79e" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 33549 } }
I20260810 00:45:30.599355 3168 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae090afa015d4aceb8e0d3db8a66e79e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:45:30.599493 3168 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae090afa015d4aceb8e0d3db8a66e79e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:45:30.599668 3168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae090afa015d4aceb8e0d3db8a66e79e [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:30.601689 3168 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae090afa015d4aceb8e0d3db8a66e79e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae090afa015d4aceb8e0d3db8a66e79e" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 33549 } }
I20260810 00:45:30.602181 3168 leader_election.cc:302] T 00000000000000000000000000000000 P ae090afa015d4aceb8e0d3db8a66e79e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae090afa015d4aceb8e0d3db8a66e79e; no voters:
I20260810 00:45:30.602775 3168 leader_election.cc:288] T 00000000000000000000000000000000 P ae090afa015d4aceb8e0d3db8a66e79e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:45:30.603005 3177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae090afa015d4aceb8e0d3db8a66e79e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:45:30.603717 3177 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae090afa015d4aceb8e0d3db8a66e79e [term 1 LEADER]: Becoming Leader. State: Replica: ae090afa015d4aceb8e0d3db8a66e79e, State: Running, Role: LEADER
I20260810 00:45:30.604317 3177 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae090afa015d4aceb8e0d3db8a66e79e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae090afa015d4aceb8e0d3db8a66e79e" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 33549 } }
I20260810 00:45:30.604933 3168 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae090afa015d4aceb8e0d3db8a66e79e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:45:30.607074 3179 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae090afa015d4aceb8e0d3db8a66e79e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae090afa015d4aceb8e0d3db8a66e79e. Latest consensus state: current_term: 1 leader_uuid: "ae090afa015d4aceb8e0d3db8a66e79e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae090afa015d4aceb8e0d3db8a66e79e" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 33549 } } }
I20260810 00:45:30.607612 3179 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae090afa015d4aceb8e0d3db8a66e79e [sys.catalog]: This master's current role is: LEADER
I20260810 00:45:30.607894 3178 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae090afa015d4aceb8e0d3db8a66e79e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae090afa015d4aceb8e0d3db8a66e79e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae090afa015d4aceb8e0d3db8a66e79e" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 33549 } } }
I20260810 00:45:30.608577 3178 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae090afa015d4aceb8e0d3db8a66e79e [sys.catalog]: This master's current role is: LEADER
I20260810 00:45:30.609411 3187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:45:30.614041 3187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:45:30.621894 3187 catalog_manager.cc:1347] Generated new cluster ID: 62fafb03e000494d9fc455554bafb53a
I20260810 00:45:30.622004 3187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:45:30.638280 3187 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:45:30.638690 3187 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:45:30.646865 3187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae090afa015d4aceb8e0d3db8a66e79e: Generated new TSK 0
I20260810 00:45:30.647857 3187 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 00:45:30.664279 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.1:0
--local_ip_for_outbound_sockets=127.28.6.1
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:33549
--builtin_ntp_servers=127.28.6.20:42745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 00:45:31.019007 3206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:31.019378 3206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:31.019541 3206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:31.026965 3206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:31.027118 3206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:31.027309 3206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.1
I20260810 00:45:31.051961 3206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:42745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:33549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.1
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:31.055846 3206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:31.060218 3206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:31.106945 3214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:31.106801 3215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:31.106849 3221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:31.109637 3206 server_base.cc:1034] running on GCE node
I20260810 00:45:31.110988 3206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:31.112761 3206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:31.114283 3206 hybrid_clock.cc:648] HybridClock initialized: now 1786322731114165 us; error 109 us; skew 500 ppm
I20260810 00:45:31.114982 3206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 00:45:31.119547 3206 webserver.cc:460] Webserver started at http://127.28.6.1:44493/ using document root <none> and password file <none>
I20260810 00:45:31.120978 3206 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:31.121290 3206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:31.121979 3206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:31.127940 3206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-0/data/instance:
uuid: "fd59a0d4bf0d479a9c4d1176223acddd"
format_stamp: "Formatted at 2026-08-10 00:45:31 on dist-test-slave-1nl6"
server_key: "7f8cd9a19f61bfd2d9a4447a34cacc48"
server_key_iv: "8df8eb86fd258395fbb3db47020d4a2b"
server_key_version: "encryptionkey@0"
I20260810 00:45:31.129456 3206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance:
uuid: "fd59a0d4bf0d479a9c4d1176223acddd"
format_stamp: "Formatted at 2026-08-10 00:45:31 on dist-test-slave-1nl6"
server_key: "7f8cd9a19f61bfd2d9a4447a34cacc48"
server_key_iv: "8df8eb86fd258395fbb3db47020d4a2b"
server_key_version: "encryptionkey@0"
I20260810 00:45:31.140362 3206 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.010s sys 0.001s
I20260810 00:45:31.148238 3230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:31.151340 3206 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.005s sys 0.001s
I20260810 00:45:31.151800 3206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-0/wal
uuid: "fd59a0d4bf0d479a9c4d1176223acddd"
format_stamp: "Formatted at 2026-08-10 00:45:31 on dist-test-slave-1nl6"
server_key: "7f8cd9a19f61bfd2d9a4447a34cacc48"
server_key_iv: "8df8eb86fd258395fbb3db47020d4a2b"
server_key_version: "encryptionkey@0"
I20260810 00:45:31.152261 3206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:31.164085 3206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:31.165617 3206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:31.166055 3206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:31.167904 3206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:45:31.170709 3206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:45:31.170929 3206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:31.171119 3206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:45:31.171344 3206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:31.369415 3206 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.1:36457
I20260810 00:45:31.369562 3388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.1:36457 every 8 connection(s)
I20260810 00:45:31.371840 3206 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
I20260810 00:45:31.376950 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 3206
I20260810 00:45:31.377419 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance
I20260810 00:45:31.379364 28696 external_mini_cluster.cc:1428] Setting key 55a6f38bb54b95f8f38e6e501ee0e662
I20260810 00:45:31.386484 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.2:0
--local_ip_for_outbound_sockets=127.28.6.2
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:33549
--builtin_ntp_servers=127.28.6.20:42745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 00:45:31.390002 3394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:31.376761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:46787 (local address 127.28.6.62:33549)
0810 00:45:31.377836 (+ 1075us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:31.377866 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:31.380878 (+ 3012us) server_negotiation.cc:407] Connection header received
0810 00:45:31.385570 (+ 4692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:31.385574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:31.385726 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:31.385854 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:31.388695 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:31.388704 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:31.388709 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:45:31.388737 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:31.388810 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:45:31.388832 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:31.388980 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:31.389405 (+ 425us) server_negotiation.cc:299] Negotiation successful
0810 00:45:31.389492 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":832,"thread_start_us":472,"threads_started":1}
I20260810 00:45:31.393921 3395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:31.377797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:33549 (local address 127.28.6.1:46787)
0810 00:45:31.380581 (+ 2784us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:31.380674 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:31.385026 (+ 4352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:31.386479 (+ 1453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:31.386509 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:31.386765 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:31.387962 (+ 1197us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 00:45:31.387977 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:31.388084 (+ 107us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 00:45:31.388412 (+ 328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:31.388979 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:31.389004 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:31.389025 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 00:45:31.389644 (+ 619us) client_negotiation.cc:240] Negotiation successful
0810 00:45:31.390409 (+ 765us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1323,"thread_start_us":799,"threads_started":1}
I20260810 00:45:31.396529 3389 heartbeater.cc:347] Connected to a master server at 127.28.6.62:33549
I20260810 00:45:31.397320 3389 heartbeater.cc:464] Registering TS with master...
I20260810 00:45:31.399019 3389 heartbeater.cc:511] Master 127.28.6.62:33549 requested a full tablet report, sending...
I20260810 00:45:31.403504 3120 ts_manager.cc:194] Registered new tserver with Master: fd59a0d4bf0d479a9c4d1176223acddd (127.28.6.1:36457)
W20260810 00:45:32.033885 3400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:32.034646 3400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:32.034832 3400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:32.048394 3400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:32.048590 3400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:32.048826 3400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.2
I20260810 00:45:32.070195 3400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:42745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:33549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.2
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:32.073493 3400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:32.078081 3400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:32.103065 3413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:32.102988 3411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:32.103236 3409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:32.109055 3400 server_base.cc:1034] running on GCE node
I20260810 00:45:32.110455 3400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:32.112277 3400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:32.113656 3400 hybrid_clock.cc:648] HybridClock initialized: now 1786322732113575 us; error 85 us; skew 500 ppm
I20260810 00:45:32.114342 3400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 00:45:32.118320 3400 webserver.cc:460] Webserver started at http://127.28.6.2:44621/ using document root <none> and password file <none>
I20260810 00:45:32.119554 3400 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:32.119766 3400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:32.120326 3400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:32.125154 3400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-1/data/instance:
uuid: "65a0873e45c34a4ea4660b25e3c37f2f"
format_stamp: "Formatted at 2026-08-10 00:45:32 on dist-test-slave-1nl6"
server_key: "f2bb39ab27c5d18dcd3aa7ccf397f899"
server_key_iv: "bef70093db66e7a51654a501f87fc70e"
server_key_version: "encryptionkey@0"
I20260810 00:45:32.126609 3400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance:
uuid: "65a0873e45c34a4ea4660b25e3c37f2f"
format_stamp: "Formatted at 2026-08-10 00:45:32 on dist-test-slave-1nl6"
server_key: "f2bb39ab27c5d18dcd3aa7ccf397f899"
server_key_iv: "bef70093db66e7a51654a501f87fc70e"
server_key_version: "encryptionkey@0"
I20260810 00:45:32.135697 3400 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260810 00:45:32.144071 3421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:32.148000 3400 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.006s sys 0.000s
I20260810 00:45:32.148339 3400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-1/wal
uuid: "65a0873e45c34a4ea4660b25e3c37f2f"
format_stamp: "Formatted at 2026-08-10 00:45:32 on dist-test-slave-1nl6"
server_key: "f2bb39ab27c5d18dcd3aa7ccf397f899"
server_key_iv: "bef70093db66e7a51654a501f87fc70e"
server_key_version: "encryptionkey@0"
I20260810 00:45:32.148945 3400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:32.166591 3400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:32.168156 3400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:32.168570 3400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:32.170066 3400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:45:32.172619 3400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:45:32.172782 3400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:32.172945 3400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:45:32.173046 3400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:32.234179 3400 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.2:44935
I20260810 00:45:32.234274 3582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.2:44935 every 8 connection(s)
I20260810 00:45:32.236248 3400 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
I20260810 00:45:32.237058 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 3400
I20260810 00:45:32.237361 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance
I20260810 00:45:32.238366 28696 external_mini_cluster.cc:1428] Setting key d89113810deffba7e7108de6d9bdd2b3
I20260810 00:45:32.244537 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.3:0
--local_ip_for_outbound_sockets=127.28.6.3
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:33549
--builtin_ntp_servers=127.28.6.20:42745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 00:45:32.251696 3586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:32.242261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:49163 (local address 127.28.6.62:33549)
0810 00:45:32.244752 (+ 2491us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:32.244773 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:32.244813 (+ 40us) server_negotiation.cc:407] Connection header received
0810 00:45:32.247563 (+ 2750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:32.247580 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:32.247801 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:32.248080 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:32.250018 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:32.250039 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:32.250053 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:45:32.250115 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:32.250240 (+ 125us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:45:32.250264 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:32.250530 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:32.250752 (+ 222us) server_negotiation.cc:299] Negotiation successful
0810 00:45:32.250882 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2237,"thread_start_us":2059,"threads_started":1}
I20260810 00:45:32.252472 3587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:32.242025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:33549 (local address 127.28.6.2:49163)
0810 00:45:32.244167 (+ 2142us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:32.244266 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:32.247163 (+ 2897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:32.248295 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:32.248311 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:32.248456 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:32.249584 (+ 1128us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 00:45:32.249591 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:32.249648 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 00:45:32.249771 (+ 123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:32.250437 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:32.250449 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:32.250455 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 00:45:32.250784 (+ 329us) client_negotiation.cc:240] Negotiation successful
0810 00:45:32.251109 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":942,"thread_start_us":558,"threads_started":1}
I20260810 00:45:32.254592 3583 heartbeater.cc:347] Connected to a master server at 127.28.6.62:33549
I20260810 00:45:32.255070 3583 heartbeater.cc:464] Registering TS with master...
I20260810 00:45:32.255934 3583 heartbeater.cc:511] Master 127.28.6.62:33549 requested a full tablet report, sending...
I20260810 00:45:32.257983 3120 ts_manager.cc:194] Registered new tserver with Master: 65a0873e45c34a4ea4660b25e3c37f2f (127.28.6.2:44935)
I20260810 00:45:32.409571 3389 heartbeater.cc:503] Master 127.28.6.62:33549 was elected leader, sending a full tablet report...
W20260810 00:45:32.825747 3589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:32.826422 3589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:32.826722 3589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:32.840811 3589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:32.841095 3589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:32.841363 3589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.3
I20260810 00:45:32.859102 3589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:42745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:33549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.3
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:32.862017 3589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:32.866034 3589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:32.891808 3595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:32.891696 3596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:32.891732 3598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:32.893898 3589 server_base.cc:1034] running on GCE node
I20260810 00:45:32.895501 3589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:32.897403 3589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:32.898815 3589 hybrid_clock.cc:648] HybridClock initialized: now 1786322732898721 us; error 94 us; skew 500 ppm
I20260810 00:45:32.899544 3589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 00:45:32.904410 3589 webserver.cc:460] Webserver started at http://127.28.6.3:45515/ using document root <none> and password file <none>
I20260810 00:45:32.906224 3589 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:32.906533 3589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:32.907434 3589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:32.915517 3589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-2/data/instance:
uuid: "bf9a09ad3d1848f4a560cdd14781345c"
format_stamp: "Formatted at 2026-08-10 00:45:32 on dist-test-slave-1nl6"
server_key: "2a2c91f65d73d22e878525e51a70afa5"
server_key_iv: "ff8f8be00926f96229becf74a6ec960e"
server_key_version: "encryptionkey@0"
I20260810 00:45:32.917747 3589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance:
uuid: "bf9a09ad3d1848f4a560cdd14781345c"
format_stamp: "Formatted at 2026-08-10 00:45:32 on dist-test-slave-1nl6"
server_key: "2a2c91f65d73d22e878525e51a70afa5"
server_key_iv: "ff8f8be00926f96229becf74a6ec960e"
server_key_version: "encryptionkey@0"
I20260810 00:45:32.930176 3589 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.013s sys 0.000s
I20260810 00:45:32.937942 3612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:32.941097 3589 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260810 00:45:32.941411 3589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-2/wal
uuid: "bf9a09ad3d1848f4a560cdd14781345c"
format_stamp: "Formatted at 2026-08-10 00:45:32 on dist-test-slave-1nl6"
server_key: "2a2c91f65d73d22e878525e51a70afa5"
server_key_iv: "ff8f8be00926f96229becf74a6ec960e"
server_key_version: "encryptionkey@0"
I20260810 00:45:32.941800 3589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:32.954571 3589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:32.956173 3589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:32.956676 3589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:32.958774 3589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:45:32.961874 3589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:45:32.962108 3589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:32.962445 3589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:45:32.962605 3589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:33.032104 3589 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.3:45959
I20260810 00:45:33.032205 3788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.3:45959 every 8 connection(s)
I20260810 00:45:33.035056 3589 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
I20260810 00:45:33.038317 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 3589
I20260810 00:45:33.038543 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance
I20260810 00:45:33.039620 28696 external_mini_cluster.cc:1428] Setting key 0006bbdc7759f804adaf0fcf305a858f
I20260810 00:45:33.054641 3794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:33.042319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:51247 (local address 127.28.6.62:33549)
0810 00:45:33.043357 (+ 1038us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:33.043362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:33.047387 (+ 4025us) server_negotiation.cc:407] Connection header received
0810 00:45:33.050253 (+ 2866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:33.050262 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:33.050373 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:33.050601 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:33.053117 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:33.053127 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:33.053135 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:45:33.053158 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:33.053221 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:45:33.053247 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:33.053529 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:33.053949 (+ 420us) server_negotiation.cc:299] Negotiation successful
0810 00:45:33.054063 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":455,"threads_started":1}
I20260810 00:45:33.056171 3796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:33.044136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:33549 (local address 127.28.6.3:51247)
0810 00:45:33.047188 (+ 3052us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:33.047242 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:33.049765 (+ 2523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:33.050994 (+ 1229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:33.051009 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:33.051141 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:33.052549 (+ 1408us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 00:45:33.052559 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:33.052669 (+ 110us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 00:45:33.052882 (+ 213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:33.053434 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:33.053482 (+ 48us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:33.053503 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 00:45:33.054063 (+ 560us) client_negotiation.cc:240] Negotiation successful
0810 00:45:33.054549 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2247,"thread_start_us":1846,"threads_started":1}
I20260810 00:45:33.058229 3791 heartbeater.cc:347] Connected to a master server at 127.28.6.62:33549
I20260810 00:45:33.058816 3791 heartbeater.cc:464] Registering TS with master...
I20260810 00:45:33.060035 3791 heartbeater.cc:511] Master 127.28.6.62:33549 requested a full tablet report, sending...
I20260810 00:45:33.062656 3120 ts_manager.cc:194] Registered new tserver with Master: bf9a09ad3d1848f4a560cdd14781345c (127.28.6.3:45959)
I20260810 00:45:33.073253 28696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 00:45:33.085954 3794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:33.081192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33824 (local address 127.28.6.62:33549)
0810 00:45:33.081630 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:33.081653 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:33.082348 (+ 695us) server_negotiation.cc:407] Connection header received
0810 00:45:33.082633 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:33.082643 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:33.082794 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:33.083026 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:33.084720 (+ 1694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:33.084727 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:33.084736 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:45:33.084761 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:33.084852 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:45:33.084868 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:33.085163 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:33.085281 (+ 118us) server_negotiation.cc:299] Negotiation successful
0810 00:45:33.085355 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260810 00:45:33.092523 3120 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33824:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 00:45:33.107077 3120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 00:45:33.163028 3814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:33.130300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:36457 (local address 127.0.0.1:36800)
0810 00:45:33.154010 (+ 23710us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:33.154100 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:33.154443 (+ 343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:33.156949 (+ 2506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:33.156977 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:33.157215 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:33.159256 (+ 2041us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 00:45:33.159269 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:33.159348 (+ 79us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 00:45:33.159545 (+ 197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:33.161266 (+ 1721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:33.161279 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:33.161298 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 00:45:33.161758 (+ 460us) client_negotiation.cc:240] Negotiation successful
0810 00:45:33.161948 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23205,"thread_start_us":28591,"threads_started":1}
I20260810 00:45:33.163241 3810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:33.130952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36800 (local address 127.28.6.1:36457)
0810 00:45:33.152162 (+ 21210us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:33.152222 (+ 60us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:33.154849 (+ 2627us) server_negotiation.cc:407] Connection header received
0810 00:45:33.155031 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:33.155052 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:33.155672 (+ 620us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:33.155976 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:33.159940 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:33.159970 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:33.160011 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:45:33.160059 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:33.160216 (+ 157us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:45:33.160279 (+ 63us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:33.161229 (+ 950us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:33.161705 (+ 476us) server_negotiation.cc:299] Negotiation successful
0810 00:45:33.161912 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20748,"thread_start_us":21250,"threads_started":1}
I20260810 00:45:33.166153 3812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:33.131222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35002 (local address 127.28.6.3:45959)
0810 00:45:33.145660 (+ 14438us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:33.145689 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:33.156254 (+ 10565us) server_negotiation.cc:407] Connection header received
0810 00:45:33.158099 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:33.158124 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:33.158709 (+ 585us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:33.159108 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:33.162732 (+ 3624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:33.162757 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:33.162796 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:45:33.162873 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:33.163051 (+ 178us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:45:33.163099 (+ 48us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:33.164419 (+ 1320us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:33.164600 (+ 181us) server_negotiation.cc:299] Negotiation successful
0810 00:45:33.164940 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13973,"thread_start_us":18906,"threads_started":1}
I20260810 00:45:33.166209 3817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:33.130766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:45959 (local address 127.0.0.1:35002)
0810 00:45:33.155916 (+ 25150us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:33.155981 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:33.157714 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:33.159560 (+ 1846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:33.159573 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:33.159628 (+ 55us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:33.162221 (+ 2593us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 00:45:33.162235 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:33.162278 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 00:45:33.162368 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:33.163958 (+ 1590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:33.163979 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:33.163995 (+ 16us) client_negotiation.cc:769] Sending connection context
0810 00:45:33.164249 (+ 254us) client_negotiation.cc:240] Negotiation successful
0810 00:45:33.164351 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24049,"thread_start_us":22459,"threads_started":1}
I20260810 00:45:33.168378 3808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:33.144444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:44935 (local address 127.0.0.1:49284)
0810 00:45:33.159491 (+ 15047us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:33.159543 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:33.159832 (+ 289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:33.161421 (+ 1589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:33.161435 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:33.161477 (+ 42us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:33.165595 (+ 4118us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 00:45:33.165608 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:33.165650 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 00:45:33.165742 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:33.166938 (+ 1196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:33.166951 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:33.166963 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 00:45:33.167281 (+ 318us) client_negotiation.cc:240] Negotiation successful
0810 00:45:33.167374 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14786,"thread_start_us":10148,"threads_started":1}
I20260810 00:45:33.169554 3816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:33.144498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49284 (local address 127.28.6.2:44935)
0810 00:45:33.157235 (+ 12737us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:33.157262 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:33.160077 (+ 2815us) server_negotiation.cc:407] Connection header received
0810 00:45:33.160298 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:33.160324 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:33.160778 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:33.162105 (+ 1327us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:33.166187 (+ 4082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:33.166209 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:33.166239 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:45:33.166280 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:33.166409 (+ 129us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:45:33.166457 (+ 48us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:33.166807 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:33.168729 (+ 1922us) server_negotiation.cc:299] Negotiation successful
0810 00:45:33.168908 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12362,"thread_start_us":12750,"threads_started":1}
I20260810 00:45:33.171938 3294 tablet_service.cc:1467] Processing CreateTablet for tablet eec1d20a08ea4b9c90d585c35886cb20 (DEFAULT_TABLE table=test-table [id=113d65e494e94b49893cac7866aa65de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:33.173846 3294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eec1d20a08ea4b9c90d585c35886cb20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:33.174861 3691 tablet_service.cc:1467] Processing CreateTablet for tablet eec1d20a08ea4b9c90d585c35886cb20 (DEFAULT_TABLE table=test-table [id=113d65e494e94b49893cac7866aa65de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:33.177644 3691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eec1d20a08ea4b9c90d585c35886cb20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:33.177148 3486 tablet_service.cc:1467] Processing CreateTablet for tablet eec1d20a08ea4b9c90d585c35886cb20 (DEFAULT_TABLE table=test-table [id=113d65e494e94b49893cac7866aa65de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:33.180368 3486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eec1d20a08ea4b9c90d585c35886cb20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:33.192306 3820 tablet_bootstrap.cc:492] T eec1d20a08ea4b9c90d585c35886cb20 P fd59a0d4bf0d479a9c4d1176223acddd: Bootstrap starting.
I20260810 00:45:33.199219 3820 tablet_bootstrap.cc:654] T eec1d20a08ea4b9c90d585c35886cb20 P fd59a0d4bf0d479a9c4d1176223acddd: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:33.200898 3820 log.cc:824] T eec1d20a08ea4b9c90d585c35886cb20 P fd59a0d4bf0d479a9c4d1176223acddd: Log is configured to *not* fsync() on all Append() calls
I20260810 00:45:33.205582 3820 tablet_bootstrap.cc:492] T eec1d20a08ea4b9c90d585c35886cb20 P fd59a0d4bf0d479a9c4d1176223acddd: No bootstrap required, opened a new log
I20260810 00:45:33.206050 3820 ts_tablet_manager.cc:1397] T eec1d20a08ea4b9c90d585c35886cb20 P fd59a0d4bf0d479a9c4d1176223acddd: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.012s
I20260810 00:45:33.208797 3824 tablet_bootstrap.cc:492] T eec1d20a08ea4b9c90d585c35886cb20 P bf9a09ad3d1848f4a560cdd14781345c: Bootstrap starting.
I20260810 00:45:33.211812 3825 tablet_bootstrap.cc:492] T eec1d20a08ea4b9c90d585c35886cb20 P 65a0873e45c34a4ea4660b25e3c37f2f: Bootstrap starting.
I20260810 00:45:33.216174 3824 tablet_bootstrap.cc:654] T eec1d20a08ea4b9c90d585c35886cb20 P bf9a09ad3d1848f4a560cdd14781345c: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:33.217609 3825 tablet_bootstrap.cc:654] T eec1d20a08ea4b9c90d585c35886cb20 P 65a0873e45c34a4ea4660b25e3c37f2f: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:33.217607 3824 log.cc:824] T eec1d20a08ea4b9c90d585c35886cb20 P bf9a09ad3d1848f4a560cdd14781345c: Log is configured to *not* fsync() on all Append() calls
I20260810 00:45:33.219537 3825 log.cc:824] T eec1d20a08ea4b9c90d585c35886cb20 P 65a0873e45c34a4ea4660b25e3c37f2f: Log is configured to *not* fsync() on all Append() calls
I20260810 00:45:33.223088 3824 tablet_bootstrap.cc:492] T eec1d20a08ea4b9c90d585c35886cb20 P bf9a09ad3d1848f4a560cdd14781345c: No bootstrap required, opened a new log
I20260810 00:45:33.223492 3824 ts_tablet_manager.cc:1397] T eec1d20a08ea4b9c90d585c35886cb20 P bf9a09ad3d1848f4a560cdd14781345c: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20260810 00:45:33.223714 3820 raft_consensus.cc:348] T eec1d20a08ea4b9c90d585c35886cb20 P fd59a0d4bf0d479a9c4d1176223acddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd59a0d4bf0d479a9c4d1176223acddd" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 36457 } } peers { permanent_uuid: "bf9a09ad3d1848f4a560cdd14781345c" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45959 } } peers { permanent_uuid: "65a0873e45c34a4ea4660b25e3c37f2f" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 44935 } }
I20260810 00:45:33.224575 3820 raft_consensus.cc:374] T eec1d20a08ea4b9c90d585c35886cb20 P fd59a0d4bf0d479a9c4d1176223acddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:33.224753 3820 raft_consensus.cc:729] T eec1d20a08ea4b9c90d585c35886cb20 P fd59a0d4bf0d479a9c4d1176223acddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd59a0d4bf0d479a9c4d1176223acddd, State: Initialized, Role: FOLLOWER
I20260810 00:45:33.225015 3825 tablet_bootstrap.cc:492] T eec1d20a08ea4b9c90d585c35886cb20 P 65a0873e45c34a4ea4660b25e3c37f2f: No bootstrap required, opened a new log
I20260810 00:45:33.225442 3825 ts_tablet_manager.cc:1397] T eec1d20a08ea4b9c90d585c35886cb20 P 65a0873e45c34a4ea4660b25e3c37f2f: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.003s
I20260810 00:45:33.226043 3820 consensus_queue.cc:260] T eec1d20a08ea4b9c90d585c35886cb20 P fd59a0d4bf0d479a9c4d1176223acddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd59a0d4bf0d479a9c4d1176223acddd" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 36457 } } peers { permanent_uuid: "bf9a09ad3d1848f4a560cdd14781345c" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45959 } } peers { permanent_uuid: "65a0873e45c34a4ea4660b25e3c37f2f" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 44935 } }
I20260810 00:45:33.230484 3820 ts_tablet_manager.cc:1428] T eec1d20a08ea4b9c90d585c35886cb20 P fd59a0d4bf0d479a9c4d1176223acddd: Time spent starting tablet: real 0.024s user 0.018s sys 0.002s
I20260810 00:45:33.242754 3825 raft_consensus.cc:348] T eec1d20a08ea4b9c90d585c35886cb20 P 65a0873e45c34a4ea4660b25e3c37f2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd59a0d4bf0d479a9c4d1176223acddd" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 36457 } } peers { permanent_uuid: "bf9a09ad3d1848f4a560cdd14781345c" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45959 } } peers { permanent_uuid: "65a0873e45c34a4ea4660b25e3c37f2f" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 44935 } }
I20260810 00:45:33.243489 3825 raft_consensus.cc:374] T eec1d20a08ea4b9c90d585c35886cb20 P 65a0873e45c34a4ea4660b25e3c37f2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:33.243652 3825 raft_consensus.cc:729] T eec1d20a08ea4b9c90d585c35886cb20 P 65a0873e45c34a4ea4660b25e3c37f2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65a0873e45c34a4ea4660b25e3c37f2f, State: Initialized, Role: FOLLOWER
I20260810 00:45:33.243234 3824 raft_consensus.cc:348] T eec1d20a08ea4b9c90d585c35886cb20 P bf9a09ad3d1848f4a560cdd14781345c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd59a0d4bf0d479a9c4d1176223acddd" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 36457 } } peers { permanent_uuid: "bf9a09ad3d1848f4a560cdd14781345c" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45959 } } peers { permanent_uuid: "65a0873e45c34a4ea4660b25e3c37f2f" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 44935 } }
I20260810 00:45:33.244110 3824 raft_consensus.cc:374] T eec1d20a08ea4b9c90d585c35886cb20 P bf9a09ad3d1848f4a560cdd14781345c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:33.244333 3824 raft_consensus.cc:729] T eec1d20a08ea4b9c90d585c35886cb20 P bf9a09ad3d1848f4a560cdd14781345c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9a09ad3d1848f4a560cdd14781345c, State: Initialized, Role: FOLLOWER
I20260810 00:45:33.244755 3825 consensus_queue.cc:260] T eec1d20a08ea4b9c90d585c35886cb20 P 65a0873e45c34a4ea4660b25e3c37f2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd59a0d4bf0d479a9c4d1176223acddd" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 36457 } } peers { permanent_uuid: "bf9a09ad3d1848f4a560cdd14781345c" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45959 } } peers { permanent_uuid: "65a0873e45c34a4ea4660b25e3c37f2f" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 44935 } }
I20260810 00:45:33.245422 3824 consensus_queue.cc:260] T eec1d20a08ea4b9c90d585c35886cb20 P bf9a09ad3d1848f4a560cdd14781345c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd59a0d4bf0d479a9c4d1176223acddd" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 36457 } } peers { permanent_uuid: "bf9a09ad3d1848f4a560cdd14781345c" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45959 } } peers { permanent_uuid: "65a0873e45c34a4ea4660b25e3c37f2f" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 44935 } }
I20260810 00:45:33.246693 3583 heartbeater.cc:503] Master 127.28.6.62:33549 was elected leader, sending a full tablet report...
I20260810 00:45:33.250299 3825 ts_tablet_manager.cc:1428] T eec1d20a08ea4b9c90d585c35886cb20 P 65a0873e45c34a4ea4660b25e3c37f2f: Time spent starting tablet: real 0.025s user 0.022s sys 0.000s
I20260810 00:45:33.251670 3791 heartbeater.cc:503] Master 127.28.6.62:33549 was elected leader, sending a full tablet report...
I20260810 00:45:33.253047 3824 ts_tablet_manager.cc:1428] T eec1d20a08ea4b9c90d585c35886cb20 P bf9a09ad3d1848f4a560cdd14781345c: Time spent starting tablet: real 0.029s user 0.021s sys 0.005s
I20260810 00:45:33.265976 3830 raft_consensus.cc:482] T eec1d20a08ea4b9c90d585c35886cb20 P bf9a09ad3d1848f4a560cdd14781345c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:45:33.266546 3830 raft_consensus.cc:504] T eec1d20a08ea4b9c90d585c35886cb20 P bf9a09ad3d1848f4a560cdd14781345c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd59a0d4bf0d479a9c4d1176223acddd" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 36457 } } peers { permanent_uuid: "bf9a09ad3d1848f4a560cdd14781345c" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45959 } } peers { permanent_uuid: "65a0873e45c34a4ea4660b25e3c37f2f" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 44935 } }
I20260810 00:45:33.269531 3830 leader_election.cc:288] T eec1d20a08ea4b9c90d585c35886cb20 P bf9a09ad3d1848f4a560cdd14781345c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd59a0d4bf0d479a9c4d1176223acddd (127.28.6.1:36457), 65a0873e45c34a4ea4660b25e3c37f2f (127.28.6.2:44935)
I20260810 00:45:33.270601 3828 raft_consensus.cc:482] T eec1d20a08ea4b9c90d585c35886cb20 P fd59a0d4bf0d479a9c4d1176223acddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:45:33.271166 3828 raft_consensus.cc:504] T eec1d20a08ea4b9c90d585c35886cb20 P fd59a0d4bf0d479a9c4d1176223acddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd59a0d4bf0d479a9c4d1176223acddd" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 36457 } } peers { permanent_uuid: "bf9a09ad3d1848f4a560cdd14781345c" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45959 } } peers { permanent_uuid: "65a0873e45c34a4ea4660b25e3c37f2f" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 44935 } }
I20260810 00:45:33.277266 3816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:33.271003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:58525 (local address 127.28.6.2:44935)
0810 00:45:33.271340 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:33.271348 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:33.272635 (+ 1287us) server_negotiation.cc:407] Connection header received
0810 00:45:33.273113 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:33.273120 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:33.273219 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:33.273366 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:33.275482 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:33.275489 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:33.275494 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:45:33.275527 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:33.275596 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:45:33.275656 (+ 60us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:33.275866 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:33.276419 (+ 553us) server_negotiation.cc:299] Negotiation successful
0810 00:45:33.276577 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260810 00:45:33.277662 3810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:33.269927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:41191 (local address 127.28.6.1:36457)
0810 00:45:33.270426 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:33.270447 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:33.271033 (+ 586us) server_negotiation.cc:407] Connection header received
0810 00:45:33.271198 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:33.271209 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:33.271391 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:33.271699 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:33.274204 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:33.274212 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:33.274226 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:45:33.274263 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:33.274353 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:45:33.274954 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:33.275244 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:33.276697 (+ 1453us) server_negotiation.cc:299] Negotiation successful
0810 00:45:33.276848 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292}
I20260810 00:45:33.276449 3796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:33.270026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:36457 (local address 127.28.6.3:41191)
0810 00:45:33.270386 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:33.270423 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:33.270646 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:33.272125 (+ 1479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:33.272138 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:33.272218 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:33.273644 (+ 1426us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 00:45:33.273653 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:33.273727 (+ 74us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 00:45:33.273826 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:33.275373 (+ 1547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:33.275383 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:33.275405 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 00:45:33.275698 (+ 293us) client_negotiation.cc:240] Negotiation successful
0810 00:45:33.275831 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131}
I20260810 00:45:33.279327 3828 leader_election.cc:288] T eec1d20a08ea4b9c90d585c35886cb20 P fd59a0d4bf0d479a9c4d1176223acddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf9a09ad3d1848f4a560cdd14781345c (127.28.6.3:45959), 65a0873e45c34a4ea4660b25e3c37f2f (127.28.6.2:44935)
I20260810 00:45:33.279896 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:33.270346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:44935 (local address 127.28.6.3:58525)
0810 00:45:33.272376 (+ 2030us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:33.272420 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:33.272833 (+ 413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:33.273729 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:33.273736 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:33.273774 (+ 38us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:33.275016 (+ 1242us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 00:45:33.275031 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:33.275065 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 00:45:33.275147 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:33.276096 (+ 949us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:33.276100 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:33.276105 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 00:45:33.278643 (+ 2538us) client_negotiation.cc:240] Negotiation successful
0810 00:45:33.278754 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1796,"thread_start_us":4200,"threads_started":1}
I20260810 00:45:33.282546 3515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec1d20a08ea4b9c90d585c35886cb20" candidate_uuid: "bf9a09ad3d1848f4a560cdd14781345c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65a0873e45c34a4ea4660b25e3c37f2f" is_pre_election: true
I20260810 00:45:33.282722 3322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec1d20a08ea4b9c90d585c35886cb20" candidate_uuid: "bf9a09ad3d1848f4a560cdd14781345c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd59a0d4bf0d479a9c4d1176223acddd" is_pre_election: true
I20260810 00:45:33.283720 3515 raft_consensus.cc:2454] T eec1d20a08ea4b9c90d585c35886cb20 P 65a0873e45c34a4ea4660b25e3c37f2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf9a09ad3d1848f4a560cdd14781345c in term 0.
I20260810 00:45:33.283742 3322 raft_consensus.cc:2454] T eec1d20a08ea4b9c90d585c35886cb20 P fd59a0d4bf0d479a9c4d1176223acddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf9a09ad3d1848f4a560cdd14781345c in term 0.
I20260810 00:45:33.285231 3618 leader_election.cc:302] T eec1d20a08ea4b9c90d585c35886cb20 P bf9a09ad3d1848f4a560cdd14781345c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65a0873e45c34a4ea4660b25e3c37f2f, bf9a09ad3d1848f4a560cdd14781345c; no voters:
I20260810 00:45:33.286036 3830 raft_consensus.cc:2781] T eec1d20a08ea4b9c90d585c35886cb20 P bf9a09ad3d1848f4a560cdd14781345c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 00:45:33.286350 3830 raft_consensus.cc:482] T eec1d20a08ea4b9c90d585c35886cb20 P bf9a09ad3d1848f4a560cdd14781345c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 00:45:33.286583 3830 raft_consensus.cc:3037] T eec1d20a08ea4b9c90d585c35886cb20 P bf9a09ad3d1848f4a560cdd14781345c [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:33.288045 3812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:33.276328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:57323 (local address 127.28.6.3:45959)
0810 00:45:33.276666 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:33.276688 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:33.281769 (+ 5081us) server_negotiation.cc:407] Connection header received
0810 00:45:33.282400 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:33.282421 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:33.282547 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:33.282795 (+ 248us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:33.286071 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:33.286113 (+ 42us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:33.286123 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:45:33.286172 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:33.286254 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:45:33.286293 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:33.286511 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:33.287212 (+ 701us) server_negotiation.cc:299] Negotiation successful
0810 00:45:33.287358 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260810 00:45:33.288359 3816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:33.275886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:60103 (local address 127.28.6.2:44935)
0810 00:45:33.277774 (+ 1888us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:33.277784 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:33.282089 (+ 4305us) server_negotiation.cc:407] Connection header received
0810 00:45:33.282696 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:33.282701 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:33.282857 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:33.283068 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:33.286322 (+ 3254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:33.286351 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:33.286372 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:45:33.286409 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:33.286481 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:45:33.286528 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:33.286732 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:33.287472 (+ 740us) server_negotiation.cc:299] Negotiation successful
0810 00:45:33.287611 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1662,"thread_start_us":3023,"threads_started":1}
W20260810 00:45:33.288530 3792 tablet.cc:2369] T eec1d20a08ea4b9c90d585c35886cb20 P bf9a09ad3d1848f4a560cdd14781345c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:45:33.289510 3837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:33.275552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:45959 (local address 127.28.6.1:57323)
0810 00:45:33.281569 (+ 6017us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:33.281620 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:33.281946 (+ 326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:33.283220 (+ 1274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:33.283297 (+ 77us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:33.283358 (+ 61us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:33.284889 (+ 1531us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 00:45:33.284969 (+ 80us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:33.285008 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 00:45:33.285710 (+ 702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:33.286725 (+ 1015us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:33.286731 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:33.286736 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 00:45:33.287304 (+ 568us) client_negotiation.cc:240] Negotiation successful
0810 00:45:33.287411 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5691,"thread_start_us":5690,"threads_started":1}
I20260810 00:45:33.290251 3834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:33.275577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:44935 (local address 127.28.6.1:60103)
0810 00:45:33.281907 (+ 6330us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:33.281958 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:33.282331 (+ 373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:33.283607 (+ 1276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:33.283615 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:33.283658 (+ 43us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:33.285828 (+ 2170us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 00:45:33.285837 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:33.285878 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 00:45:33.285960 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:33.287071 (+ 1111us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:33.287076 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:33.287184 (+ 108us) client_negotiation.cc:769] Sending connection context
0810 00:45:33.288899 (+ 1715us) client_negotiation.cc:240] Negotiation successful
0810 00:45:33.289081 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6048,"mutex_wait_us":933,"thread_start_us":3289,"threads_started":1}
I20260810 00:45:33.291707 3515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec1d20a08ea4b9c90d585c35886cb20" candidate_uuid: "fd59a0d4bf0d479a9c4d1176223acddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65a0873e45c34a4ea4660b25e3c37f2f" is_pre_election: true
I20260810 00:45:33.291410 3728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec1d20a08ea4b9c90d585c35886cb20" candidate_uuid: "fd59a0d4bf0d479a9c4d1176223acddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9a09ad3d1848f4a560cdd14781345c" is_pre_election: true
I20260810 00:45:33.292219 3515 raft_consensus.cc:2454] T eec1d20a08ea4b9c90d585c35886cb20 P 65a0873e45c34a4ea4660b25e3c37f2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd59a0d4bf0d479a9c4d1176223acddd in term 0.
I20260810 00:45:33.293007 3237 leader_election.cc:302] T eec1d20a08ea4b9c90d585c35886cb20 P fd59a0d4bf0d479a9c4d1176223acddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65a0873e45c34a4ea4660b25e3c37f2f, fd59a0d4bf0d479a9c4d1176223acddd; no voters:
I20260810 00:45:33.293867 3828 raft_consensus.cc:2781] T eec1d20a08ea4b9c90d585c35886cb20 P fd59a0d4bf0d479a9c4d1176223acddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 00:45:33.294214 3828 raft_consensus.cc:482] T eec1d20a08ea4b9c90d585c35886cb20 P fd59a0d4bf0d479a9c4d1176223acddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 00:45:33.294399 3828 raft_consensus.cc:3037] T eec1d20a08ea4b9c90d585c35886cb20 P fd59a0d4bf0d479a9c4d1176223acddd [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:33.295277 3830 raft_consensus.cc:504] T eec1d20a08ea4b9c90d585c35886cb20 P bf9a09ad3d1848f4a560cdd14781345c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd59a0d4bf0d479a9c4d1176223acddd" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 36457 } } peers { permanent_uuid: "bf9a09ad3d1848f4a560cdd14781345c" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45959 } } peers { permanent_uuid: "65a0873e45c34a4ea4660b25e3c37f2f" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 44935 } }
I20260810 00:45:33.296126 3728 raft_consensus.cc:2379] T eec1d20a08ea4b9c90d585c35886cb20 P bf9a09ad3d1848f4a560cdd14781345c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fd59a0d4bf0d479a9c4d1176223acddd in current term 1: Already voted for candidate bf9a09ad3d1848f4a560cdd14781345c in this term.
I20260810 00:45:33.297521 3830 leader_election.cc:288] T eec1d20a08ea4b9c90d585c35886cb20 P bf9a09ad3d1848f4a560cdd14781345c [CANDIDATE]: Term 1 election: Requested vote from peers fd59a0d4bf0d479a9c4d1176223acddd (127.28.6.1:36457), 65a0873e45c34a4ea4660b25e3c37f2f (127.28.6.2:44935)
I20260810 00:45:33.297520 3322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec1d20a08ea4b9c90d585c35886cb20" candidate_uuid: "bf9a09ad3d1848f4a560cdd14781345c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd59a0d4bf0d479a9c4d1176223acddd"
I20260810 00:45:33.298417 3515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec1d20a08ea4b9c90d585c35886cb20" candidate_uuid: "bf9a09ad3d1848f4a560cdd14781345c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65a0873e45c34a4ea4660b25e3c37f2f"
I20260810 00:45:33.298835 3515 raft_consensus.cc:3037] T eec1d20a08ea4b9c90d585c35886cb20 P 65a0873e45c34a4ea4660b25e3c37f2f [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:33.302304 3828 raft_consensus.cc:504] T eec1d20a08ea4b9c90d585c35886cb20 P fd59a0d4bf0d479a9c4d1176223acddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd59a0d4bf0d479a9c4d1176223acddd" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 36457 } } peers { permanent_uuid: "bf9a09ad3d1848f4a560cdd14781345c" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45959 } } peers { permanent_uuid: "65a0873e45c34a4ea4660b25e3c37f2f" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 44935 } }
I20260810 00:45:33.303174 3322 raft_consensus.cc:2379] T eec1d20a08ea4b9c90d585c35886cb20 P fd59a0d4bf0d479a9c4d1176223acddd [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate bf9a09ad3d1848f4a560cdd14781345c in current term 1: Already voted for candidate fd59a0d4bf0d479a9c4d1176223acddd in this term.
I20260810 00:45:33.303966 3828 leader_election.cc:288] T eec1d20a08ea4b9c90d585c35886cb20 P fd59a0d4bf0d479a9c4d1176223acddd [CANDIDATE]: Term 1 election: Requested vote from peers bf9a09ad3d1848f4a560cdd14781345c (127.28.6.3:45959), 65a0873e45c34a4ea4660b25e3c37f2f (127.28.6.2:44935)
I20260810 00:45:33.304760 3728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec1d20a08ea4b9c90d585c35886cb20" candidate_uuid: "fd59a0d4bf0d479a9c4d1176223acddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9a09ad3d1848f4a560cdd14781345c"
I20260810 00:45:33.305347 3728 raft_consensus.cc:2379] T eec1d20a08ea4b9c90d585c35886cb20 P bf9a09ad3d1848f4a560cdd14781345c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fd59a0d4bf0d479a9c4d1176223acddd in current term 1: Already voted for candidate bf9a09ad3d1848f4a560cdd14781345c in this term.
I20260810 00:45:33.305433 3515 raft_consensus.cc:2454] T eec1d20a08ea4b9c90d585c35886cb20 P 65a0873e45c34a4ea4660b25e3c37f2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9a09ad3d1848f4a560cdd14781345c in term 1.
I20260810 00:45:33.306171 3618 leader_election.cc:302] T eec1d20a08ea4b9c90d585c35886cb20 P bf9a09ad3d1848f4a560cdd14781345c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65a0873e45c34a4ea4660b25e3c37f2f, bf9a09ad3d1848f4a560cdd14781345c; no voters: fd59a0d4bf0d479a9c4d1176223acddd
I20260810 00:45:33.306353 3514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec1d20a08ea4b9c90d585c35886cb20" candidate_uuid: "fd59a0d4bf0d479a9c4d1176223acddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65a0873e45c34a4ea4660b25e3c37f2f"
I20260810 00:45:33.306721 3830 raft_consensus.cc:2781] T eec1d20a08ea4b9c90d585c35886cb20 P bf9a09ad3d1848f4a560cdd14781345c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:45:33.306944 3514 raft_consensus.cc:2379] T eec1d20a08ea4b9c90d585c35886cb20 P 65a0873e45c34a4ea4660b25e3c37f2f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fd59a0d4bf0d479a9c4d1176223acddd in current term 1: Already voted for candidate bf9a09ad3d1848f4a560cdd14781345c in this term.
I20260810 00:45:33.307998 3830 raft_consensus.cc:686] T eec1d20a08ea4b9c90d585c35886cb20 P bf9a09ad3d1848f4a560cdd14781345c [term 1 LEADER]: Becoming Leader. State: Replica: bf9a09ad3d1848f4a560cdd14781345c, State: Running, Role: LEADER
I20260810 00:45:33.308293 3237 leader_election.cc:302] T eec1d20a08ea4b9c90d585c35886cb20 P fd59a0d4bf0d479a9c4d1176223acddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd59a0d4bf0d479a9c4d1176223acddd; no voters: 65a0873e45c34a4ea4660b25e3c37f2f, bf9a09ad3d1848f4a560cdd14781345c
I20260810 00:45:33.308992 3828 raft_consensus.cc:2726] T eec1d20a08ea4b9c90d585c35886cb20 P fd59a0d4bf0d479a9c4d1176223acddd [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 00:45:33.309062 3830 consensus_queue.cc:237] T eec1d20a08ea4b9c90d585c35886cb20 P bf9a09ad3d1848f4a560cdd14781345c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd59a0d4bf0d479a9c4d1176223acddd" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 36457 } } peers { permanent_uuid: "bf9a09ad3d1848f4a560cdd14781345c" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45959 } } peers { permanent_uuid: "65a0873e45c34a4ea4660b25e3c37f2f" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 44935 } }
I20260810 00:45:33.316402 3119 catalog_manager.cc:5495] T eec1d20a08ea4b9c90d585c35886cb20 P bf9a09ad3d1848f4a560cdd14781345c reported cstate change: term changed from 0 to 1, leader changed from <none> to bf9a09ad3d1848f4a560cdd14781345c (127.28.6.3). New cstate: current_term: 1 leader_uuid: "bf9a09ad3d1848f4a560cdd14781345c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd59a0d4bf0d479a9c4d1176223acddd" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 36457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf9a09ad3d1848f4a560cdd14781345c" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65a0873e45c34a4ea4660b25e3c37f2f" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 44935 } health_report { overall_health: UNKNOWN } } }
I20260810 00:45:33.360591 3812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:33.357316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35004 (local address 127.28.6.3:45959)
0810 00:45:33.357565 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:33.357570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:33.357601 (+ 31us) server_negotiation.cc:407] Connection header received
0810 00:45:33.357872 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:33.357877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:33.358021 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:33.358179 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:33.359499 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:33.359506 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:33.359523 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:45:33.359549 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:33.359597 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:45:33.359648 (+ 51us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:33.359806 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:33.359934 (+ 128us) server_negotiation.cc:299] Negotiation successful
0810 00:45:33.360013 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260810 00:45:33.373140 3514 raft_consensus.cc:1261] T eec1d20a08ea4b9c90d585c35886cb20 P 65a0873e45c34a4ea4660b25e3c37f2f [term 1 FOLLOWER]: Refusing update from remote peer bf9a09ad3d1848f4a560cdd14781345c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:45:33.373382 3322 raft_consensus.cc:1261] T eec1d20a08ea4b9c90d585c35886cb20 P fd59a0d4bf0d479a9c4d1176223acddd [term 1 FOLLOWER]: Refusing update from remote peer bf9a09ad3d1848f4a560cdd14781345c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:45:33.374516 3839 consensus_queue.cc:1035] T eec1d20a08ea4b9c90d585c35886cb20 P bf9a09ad3d1848f4a560cdd14781345c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd59a0d4bf0d479a9c4d1176223acddd" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:45:33.375111 3830 consensus_queue.cc:1035] T eec1d20a08ea4b9c90d585c35886cb20 P bf9a09ad3d1848f4a560cdd14781345c [LEADER]: Connected to new peer: Peer: permanent_uuid: "65a0873e45c34a4ea4660b25e3c37f2f" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 44935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 00:45:33.376708 3392 tablet.cc:2369] T eec1d20a08ea4b9c90d585c35886cb20 P fd59a0d4bf0d479a9c4d1176223acddd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:45:33.398383 3845 mvcc.cc:205] Tried to move back new op lower bound from 7316777915882082304 to 7316777915641417728. Current Snapshot: MvccSnapshot[applied={T|T < 7316777915882082304}]
I20260810 00:45:33.403004 3851 mvcc.cc:205] Tried to move back new op lower bound from 7316777915882082304 to 7316777915641417728. Current Snapshot: MvccSnapshot[applied={T|T < 7316777915882082304}]
I20260810 00:45:33.424371 3119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:45:33.424749 3119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:45:33.429982 3119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae090afa015d4aceb8e0d3db8a66e79e: Sending DeleteTablet for 3 replicas of tablet eec1d20a08ea4b9c90d585c35886cb20
I20260810 00:45:33.431743 3294 tablet_service.cc:1514] Processing DeleteTablet for tablet eec1d20a08ea4b9c90d585c35886cb20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:45:33 UTC) from {username='slave'} at 127.0.0.1:36800
I20260810 00:45:33.431775 3691 tablet_service.cc:1514] Processing DeleteTablet for tablet eec1d20a08ea4b9c90d585c35886cb20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:45:33 UTC) from {username='slave'} at 127.0.0.1:35002
I20260810 00:45:33.431983 3486 tablet_service.cc:1514] Processing DeleteTablet for tablet eec1d20a08ea4b9c90d585c35886cb20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:45:33 UTC) from {username='slave'} at 127.0.0.1:49284
I20260810 00:45:33.440989 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 3206
I20260810 00:45:33.450448 3810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:33.443539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36810 (local address 127.28.6.1:36457)
0810 00:45:33.443882 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:33.443892 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:33.446013 (+ 2121us) server_negotiation.cc:407] Connection header received
0810 00:45:33.446433 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:33.446439 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:33.446709 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:33.446942 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:33.448857 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:33.448871 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:33.448879 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:45:33.448917 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:33.448976 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:45:33.449012 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:33.449183 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:33.449721 (+ 538us) server_negotiation.cc:299] Negotiation successful
0810 00:45:33.449856 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260810 00:45:33.453075 3380 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:45:33.453292 3862 tablet_replica.cc:330] T eec1d20a08ea4b9c90d585c35886cb20 P 65a0873e45c34a4ea4660b25e3c37f2f: stopping tablet replica
I20260810 00:45:33.454082 3862 raft_consensus.cc:2229] T eec1d20a08ea4b9c90d585c35886cb20 P 65a0873e45c34a4ea4660b25e3c37f2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:45:33.455137 3862 raft_consensus.cc:2258] T eec1d20a08ea4b9c90d585c35886cb20 P 65a0873e45c34a4ea4660b25e3c37f2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:45:33.452430 3861 tablet_replica.cc:330] T eec1d20a08ea4b9c90d585c35886cb20 P bf9a09ad3d1848f4a560cdd14781345c: stopping tablet replica
I20260810 00:45:33.457265 3861 raft_consensus.cc:2229] T eec1d20a08ea4b9c90d585c35886cb20 P bf9a09ad3d1848f4a560cdd14781345c [term 1 LEADER]: Raft consensus shutting down.
I20260810 00:45:33.458489 3861 raft_consensus.cc:2258] T eec1d20a08ea4b9c90d585c35886cb20 P bf9a09ad3d1848f4a560cdd14781345c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:45:33.463554 3862 ts_tablet_manager.cc:1905] T eec1d20a08ea4b9c90d585c35886cb20 P 65a0873e45c34a4ea4660b25e3c37f2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:45:33.465994 3861 ts_tablet_manager.cc:1905] T eec1d20a08ea4b9c90d585c35886cb20 P bf9a09ad3d1848f4a560cdd14781345c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:45:33.487349 3861 ts_tablet_manager.cc:1918] T eec1d20a08ea4b9c90d585c35886cb20 P bf9a09ad3d1848f4a560cdd14781345c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:45:33.487753 3861 log.cc:1196] T eec1d20a08ea4b9c90d585c35886cb20 P bf9a09ad3d1848f4a560cdd14781345c: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-2/wal/wals/eec1d20a08ea4b9c90d585c35886cb20
I20260810 00:45:33.488070 3862 ts_tablet_manager.cc:1918] T eec1d20a08ea4b9c90d585c35886cb20 P 65a0873e45c34a4ea4660b25e3c37f2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:45:33.488389 3862 log.cc:1196] T eec1d20a08ea4b9c90d585c35886cb20 P 65a0873e45c34a4ea4660b25e3c37f2f: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786322687226232-28696-0/minicluster-data/ts-1/wal/wals/eec1d20a08ea4b9c90d585c35886cb20
I20260810 00:45:33.489094 3861 ts_tablet_manager.cc:1939] T eec1d20a08ea4b9c90d585c35886cb20 P bf9a09ad3d1848f4a560cdd14781345c: Deleting consensus metadata
I20260810 00:45:33.489677 3862 ts_tablet_manager.cc:1939] T eec1d20a08ea4b9c90d585c35886cb20 P 65a0873e45c34a4ea4660b25e3c37f2f: Deleting consensus metadata
I20260810 00:45:33.493614 3099 catalog_manager.cc:4841] TS 65a0873e45c34a4ea4660b25e3c37f2f (127.28.6.2:44935): tablet eec1d20a08ea4b9c90d585c35886cb20 (table test-table [id=113d65e494e94b49893cac7866aa65de]) successfully deleted
I20260810 00:45:33.493546 3096 catalog_manager.cc:4841] TS bf9a09ad3d1848f4a560cdd14781345c (127.28.6.3:45959): tablet eec1d20a08ea4b9c90d585c35886cb20 (table test-table [id=113d65e494e94b49893cac7866aa65de]) successfully deleted
I20260810 00:45:33.647904 3860 tablet_replica.cc:330] T eec1d20a08ea4b9c90d585c35886cb20 P fd59a0d4bf0d479a9c4d1176223acddd: stopping tablet replica
I20260810 00:45:33.648830 3860 raft_consensus.cc:2229] T eec1d20a08ea4b9c90d585c35886cb20 P fd59a0d4bf0d479a9c4d1176223acddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:45:33.649307 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 3206
W20260810 00:45:33.717509 3098 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.6.1:36457 (error 108)
I20260810 00:45:33.719409 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 3400
I20260810 00:45:33.721724 3870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:33.720076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:36457 (local address 127.0.0.1:36820)
0810 00:45:33.721060 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:33.721307 (+ 247us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.1:36457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":738,"thread_start_us":559,"threads_started":1}
W20260810 00:45:33.722254 3098 catalog_manager.cc:4584] TS fd59a0d4bf0d479a9c4d1176223acddd (127.28.6.1:36457): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eec1d20a08ea4b9c90d585c35886cb20: Network error: Client connection negotiation failed: client connection to 127.28.6.1:36457: connect: Connection refused (error 111)
I20260810 00:45:33.737797 3816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:33.721626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49298 (local address 127.28.6.2:44935)
0810 00:45:33.721987 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:33.722008 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:33.722040 (+ 32us) server_negotiation.cc:407] Connection header received
0810 00:45:33.722230 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:33.722240 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:33.722427 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:33.722632 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:33.736628 (+ 13996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:33.736641 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:33.736645 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:45:33.736674 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:33.736728 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:45:33.736761 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:33.737047 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:33.737312 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 00:45:33.737420 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260810 00:45:33.738840 3578 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:45:33.777596 3870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:33.776627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:36457 (local address 127.0.0.1:36830)
0810 00:45:33.777071 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:33.777311 (+ 240us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.1:36457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260810 00:45:33.779819 3870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:33.779009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:36457 (local address 127.0.0.1:36842)
0810 00:45:33.779260 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:33.779502 (+ 242us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.1:36457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260810 00:45:33.841295 3870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:33.840432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:36457 (local address 127.0.0.1:36850)
0810 00:45:33.840815 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:33.840973 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.1:36457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260810 00:45:33.843143 3870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:33.842526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:36457 (local address 127.0.0.1:36866)
0810 00:45:33.842742 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:33.842875 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.1:36457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260810 00:45:33.876765 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 3400
I20260810 00:45:33.925339 3870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:33.923983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:36457 (local address 127.0.0.1:36876)
0810 00:45:33.924542 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:33.924764 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.1:36457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260810 00:45:33.927826 3870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:33.926920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:36457 (local address 127.0.0.1:36884)
0810 00:45:33.927347 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:33.927507 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.1:36457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204}
I20260810 00:45:33.935472 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 3589
I20260810 00:45:33.941488 3872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:33.936624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35008 (local address 127.28.6.3:45959)
0810 00:45:33.938048 (+ 1424us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:33.938072 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:33.938126 (+ 54us) server_negotiation.cc:407] Connection header received
0810 00:45:33.938255 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:33.938261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:33.938414 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:33.938642 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:33.940144 (+ 1502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:33.940152 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:33.940168 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:45:33.940208 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:33.940293 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:45:33.940338 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:33.940626 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:33.940712 (+ 86us) server_negotiation.cc:299] Negotiation successful
0810 00:45:33.940838 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1133,"thread_start_us":840,"threads_started":1}
I20260810 00:45:33.942958 3784 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:45:34.102367 3870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:34.101360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:36457 (local address 127.0.0.1:36896)
0810 00:45:34.101702 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:34.101957 (+ 255us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.1:36457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260810 00:45:34.104682 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 3589
I20260810 00:45:34.104839 3870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:34.103873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:36457 (local address 127.0.0.1:36898)
0810 00:45:34.104322 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:34.104513 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.1:36457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241}
I20260810 00:45:34.148561 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 3071
I20260810 00:45:34.151417 3162 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:45:34.312948 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 3071
2026-08-10T00:45:34Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (4273 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260810 00:45:34.411734 28696 mini_oidc.cc:152] Starting JWKS server
I20260810 00:45:34.418459 28696 webserver.cc:460] Webserver started at https://127.0.0.1:35177/ using document root <none> and password file <none>
I20260810 00:45:34.419214 28696 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 00:45:34.420449 28696 webserver.cc:460] Webserver started at http://127.0.0.1:41005/ using document root <none> and password file <none>
2026-08-10T00:45:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:45:34Z Disabled control of system clock
I20260810 00:45:34.472730 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:40123
--webserver_interface=127.28.6.62
--webserver_port=0
--builtin_ntp_servers=127.28.6.20:41047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:40123
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35177/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786322687226232-28696-0/testchainca.cert with env {}
W20260810 00:45:34.807497 3883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:34.807958 3883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:34.808161 3883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:34.816360 3883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:45:34.816484 3883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:34.816522 3883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:34.816592 3883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:45:34.816635 3883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 00:45:34.816689 3883 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 00:45:34.816726 3883 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35177/jwks.json
I20260810 00:45:34.826401 3883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:41047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:40123
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:40123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35177/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786322687226232-28696-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:34.827617 3883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:34.828858 3883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:34.840390 3894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:34.840446 3892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:34.840344 3896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:34.841542 3883 server_base.cc:1034] running on GCE node
I20260810 00:45:34.842398 3883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:34.843765 3883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:34.844993 3883 hybrid_clock.cc:648] HybridClock initialized: now 1786322734844959 us; error 49 us; skew 500 ppm
I20260810 00:45:34.845332 3883 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 00:45:34.847741 3883 webserver.cc:460] Webserver started at http://127.28.6.62:45269/ using document root <none> and password file <none>
I20260810 00:45:34.848395 3883 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:34.848534 3883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:34.848884 3883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:34.851907 3883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "58b79b574e9240bbbf5acf41f4c993b8"
format_stamp: "Formatted at 2026-08-10 00:45:34 on dist-test-slave-1nl6"
server_key: "b0e8d3376c5f06a6b80241542c1affe5"
server_key_iv: "921a5c10ac5a8dab28f9c1c882a4c740"
server_key_version: "encryptionkey@0"
I20260810 00:45:34.852829 3883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "58b79b574e9240bbbf5acf41f4c993b8"
format_stamp: "Formatted at 2026-08-10 00:45:34 on dist-test-slave-1nl6"
server_key: "b0e8d3376c5f06a6b80241542c1affe5"
server_key_iv: "921a5c10ac5a8dab28f9c1c882a4c740"
server_key_version: "encryptionkey@0"
I20260810 00:45:34.857962 3883 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 00:45:34.861238 3907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:34.862526 3883 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 00:45:34.862720 3883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "58b79b574e9240bbbf5acf41f4c993b8"
format_stamp: "Formatted at 2026-08-10 00:45:34 on dist-test-slave-1nl6"
server_key: "b0e8d3376c5f06a6b80241542c1affe5"
server_key_iv: "921a5c10ac5a8dab28f9c1c882a4c740"
server_key_version: "encryptionkey@0"
I20260810 00:45:34.862951 3883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:34.887508 3883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:34.888388 3883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:34.888693 3883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:34.907945 3883 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:40123
I20260810 00:45:34.907982 3971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:40123 every 8 connection(s)
I20260810 00:45:34.909253 3883 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:45:34.912073 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 3883
I20260810 00:45:34.912319 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:45:34.912993 28696 external_mini_cluster.cc:1428] Setting key 9ac2f91d46752c8c92286b7e0630d5cf
I20260810 00:45:34.913122 3972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:34.928395 3975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:34.914931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48636 (local address 127.28.6.62:40123)
0810 00:45:34.916051 (+ 1120us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:34.916065 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:34.916122 (+ 57us) server_negotiation.cc:407] Connection header received
0810 00:45:34.918217 (+ 2095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:34.918245 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:34.918827 (+ 582us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:34.919262 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:34.919818 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:34.921844 (+ 2026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:34.923214 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:34.924407 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:34.924590 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:34.925750 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:34.925760 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:34.925781 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:45:34.926500 (+ 719us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:34.926604 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:45:34.926631 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:34.926749 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:34.926934 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 00:45:34.927211 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":355,"threads_started":1}
I20260810 00:45:34.928681 3972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58b79b574e9240bbbf5acf41f4c993b8: Bootstrap starting.
I20260810 00:45:34.932456 28696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 00:45:34.932549 3972 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58b79b574e9240bbbf5acf41f4c993b8: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:34.933722 3972 log.cc:824] T 00000000000000000000000000000000 P 58b79b574e9240bbbf5acf41f4c993b8: Log is configured to *not* fsync() on all Append() calls
I20260810 00:45:34.936722 3972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58b79b574e9240bbbf5acf41f4c993b8: No bootstrap required, opened a new log
I20260810 00:45:34.943653 3972 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58b79b574e9240bbbf5acf41f4c993b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58b79b574e9240bbbf5acf41f4c993b8" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 40123 } }
I20260810 00:45:34.943948 3972 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58b79b574e9240bbbf5acf41f4c993b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:34.944020 3972 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58b79b574e9240bbbf5acf41f4c993b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58b79b574e9240bbbf5acf41f4c993b8, State: Initialized, Role: FOLLOWER
I20260810 00:45:34.944504 3972 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58b79b574e9240bbbf5acf41f4c993b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58b79b574e9240bbbf5acf41f4c993b8" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 40123 } }
I20260810 00:45:34.944692 3972 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58b79b574e9240bbbf5acf41f4c993b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:45:34.944787 3972 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58b79b574e9240bbbf5acf41f4c993b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:45:34.944944 3972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58b79b574e9240bbbf5acf41f4c993b8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:34.947405 3972 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58b79b574e9240bbbf5acf41f4c993b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58b79b574e9240bbbf5acf41f4c993b8" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 40123 } }
I20260810 00:45:34.947813 3972 leader_election.cc:302] T 00000000000000000000000000000000 P 58b79b574e9240bbbf5acf41f4c993b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58b79b574e9240bbbf5acf41f4c993b8; no voters:
I20260810 00:45:34.948319 3972 leader_election.cc:288] T 00000000000000000000000000000000 P 58b79b574e9240bbbf5acf41f4c993b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:45:34.948464 3979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58b79b574e9240bbbf5acf41f4c993b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:45:34.949200 3979 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58b79b574e9240bbbf5acf41f4c993b8 [term 1 LEADER]: Becoming Leader. State: Replica: 58b79b574e9240bbbf5acf41f4c993b8, State: Running, Role: LEADER
I20260810 00:45:34.949687 3979 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58b79b574e9240bbbf5acf41f4c993b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58b79b574e9240bbbf5acf41f4c993b8" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 40123 } }
I20260810 00:45:34.950309 3972 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58b79b574e9240bbbf5acf41f4c993b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:45:34.952785 3983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58b79b574e9240bbbf5acf41f4c993b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58b79b574e9240bbbf5acf41f4c993b8. Latest consensus state: current_term: 1 leader_uuid: "58b79b574e9240bbbf5acf41f4c993b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58b79b574e9240bbbf5acf41f4c993b8" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 40123 } } }
I20260810 00:45:34.953260 3983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58b79b574e9240bbbf5acf41f4c993b8 [sys.catalog]: This master's current role is: LEADER
I20260810 00:45:34.952936 3982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58b79b574e9240bbbf5acf41f4c993b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58b79b574e9240bbbf5acf41f4c993b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58b79b574e9240bbbf5acf41f4c993b8" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 40123 } } }
I20260810 00:45:34.954870 3982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58b79b574e9240bbbf5acf41f4c993b8 [sys.catalog]: This master's current role is: LEADER
I20260810 00:45:34.955677 3989 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:45:34.961246 3989 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:45:34.969986 3989 catalog_manager.cc:1347] Generated new cluster ID: 8c0681dc52674c80b0517ebdee1924c1
I20260810 00:45:34.970089 3989 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:45:34.988788 3989 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:45:34.990535 3989 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:45:34.998994 3989 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58b79b574e9240bbbf5acf41f4c993b8: Generated new TSK 0
I20260810 00:45:34.999734 3989 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 00:45:35.050433 4017 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260810 00:45:35.050639 4017 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 00:45:35.052887 3975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:35.028199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48652 (local address 127.28.6.62:40123)
0810 00:45:35.028407 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:35.028414 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:35.028843 (+ 429us) server_negotiation.cc:407] Connection header received
0810 00:45:35.029184 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:35.029190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:35.029301 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:35.029465 (+ 164us) server_negotiation.cc:226] Negotiated authn=JWT
0810 00:45:35.029986 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:35.031184 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:35.033084 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:35.033792 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:35.033926 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:35.034454 (+ 528us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 00:45:35.052201 (+ 17747us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35177/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 00:45:35.052570 (+ 369us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48652: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35177/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":85}
W20260810 00:45:35.053013 3975 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48652: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35177/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260810 00:45:35.054430 4016 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:45:35.028051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:40123 (local address 127.0.0.1:48652)
0810 00:45:35.028744 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:35.028760 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:35.028907 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:35.029486 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:35.029489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:35.029499 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0810 00:45:35.029838 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:35.029846 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:35.031376 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:35.031384 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:35.032935 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:35.032949 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:35.033914 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:35.033967 (+ 53us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:35.034090 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:35.034149 (+ 59us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 00:45:35.052502 (+ 18353us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35177/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 00:45:35.052769 (+ 267us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.6.62:40123: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35177/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":381,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.28.6.62:40123: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35177/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260810 00:45:35.056317 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 3883
I20260810 00:45:35.057806 3966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:45:35.152045 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 3883
2026-08-10T00:45:35Z chronyd exiting
I20260810 00:45:35.210501 28696 test_util.cc:175] -----------------------------------------------
I20260810 00:45:35.210654 28696 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786322687226232-28696-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (807 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260810 00:45:35.214468 28696 mini_oidc.cc:152] Starting JWKS server
I20260810 00:45:35.216315 28696 webserver.cc:460] Webserver started at https://127.0.0.1:36349/ using document root <none> and password file <none>
I20260810 00:45:35.216548 28696 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 00:45:35.217216 28696 webserver.cc:460] Webserver started at http://127.0.0.1:42271/ using document root <none> and password file <none>
2026-08-10T00:45:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:45:35Z Disabled control of system clock
I20260810 00:45:35.246845 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:42593
--webserver_interface=127.28.6.62
--webserver_port=0
--builtin_ntp_servers=127.28.6.20:46173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:42593
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36349/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786322687226232-28696-0/testchainca.cert with env {}
W20260810 00:45:35.571347 4027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:35.571664 4027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:35.571828 4027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:35.578948 4027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:45:35.579097 4027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:35.579180 4027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:35.579293 4027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:45:35.579370 4027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 00:45:35.579456 4027 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 00:45:35.579533 4027 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36349/jwks.json
I20260810 00:45:35.589119 4027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:46173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:42593
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:42593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36349/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786322687226232-28696-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:35.590345 4027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:35.591723 4027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:35.603158 4037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:35.603190 4040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:35.603330 4038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:35.604506 4027 server_base.cc:1034] running on GCE node
I20260810 00:45:35.605180 4027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:35.606418 4027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:35.607621 4027 hybrid_clock.cc:648] HybridClock initialized: now 1786322735607581 us; error 23 us; skew 500 ppm
I20260810 00:45:35.607972 4027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 00:45:35.610122 4027 webserver.cc:460] Webserver started at http://127.28.6.62:41639/ using document root <none> and password file <none>
I20260810 00:45:35.610796 4027 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:35.610947 4027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:35.611300 4027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:35.614149 4027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "012db187cabe47ac8ed136762bac6e3a"
format_stamp: "Formatted at 2026-08-10 00:45:35 on dist-test-slave-1nl6"
server_key: "c54c8d215612f38f71a103f39d752489"
server_key_iv: "c975c680a7cce6bdaea67d56733f27f9"
server_key_version: "encryptionkey@0"
I20260810 00:45:35.615103 4027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "012db187cabe47ac8ed136762bac6e3a"
format_stamp: "Formatted at 2026-08-10 00:45:35 on dist-test-slave-1nl6"
server_key: "c54c8d215612f38f71a103f39d752489"
server_key_iv: "c975c680a7cce6bdaea67d56733f27f9"
server_key_version: "encryptionkey@0"
I20260810 00:45:35.620280 4027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 00:45:35.623495 4053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:35.624780 4027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 00:45:35.624969 4027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "012db187cabe47ac8ed136762bac6e3a"
format_stamp: "Formatted at 2026-08-10 00:45:35 on dist-test-slave-1nl6"
server_key: "c54c8d215612f38f71a103f39d752489"
server_key_iv: "c975c680a7cce6bdaea67d56733f27f9"
server_key_version: "encryptionkey@0"
I20260810 00:45:35.625180 4027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:35.664273 4027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:35.665166 4027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:35.665465 4027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:35.683935 4027 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:42593
I20260810 00:45:35.683972 4122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:42593 every 8 connection(s)
I20260810 00:45:35.685281 4027 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:45:35.685833 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 4027
I20260810 00:45:35.686029 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:45:35.686861 28696 external_mini_cluster.cc:1428] Setting key ef66a70b7c38d9a55b8b29d9b75f0ea3
I20260810 00:45:35.689651 4123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:35.699682 4127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:35.688751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33302 (local address 127.28.6.62:42593)
0810 00:45:35.689724 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:35.689734 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:35.689805 (+ 71us) server_negotiation.cc:407] Connection header received
0810 00:45:35.691459 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:35.691479 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:35.691883 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:35.692302 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:35.692843 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:35.694118 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:35.695439 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:35.696064 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:35.696254 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:35.697429 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:35.697436 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:35.697449 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:45:35.697993 (+ 544us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:35.698063 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:45:35.698086 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:35.698217 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:35.698480 (+ 263us) server_negotiation.cc:299] Negotiation successful
0810 00:45:35.698751 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":402,"threads_started":1}
I20260810 00:45:35.700505 4123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 012db187cabe47ac8ed136762bac6e3a: Bootstrap starting.
I20260810 00:45:35.703490 28696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 00:45:35.704178 4123 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 012db187cabe47ac8ed136762bac6e3a: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:35.705200 4123 log.cc:824] T 00000000000000000000000000000000 P 012db187cabe47ac8ed136762bac6e3a: Log is configured to *not* fsync() on all Append() calls
I20260810 00:45:35.708175 4123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 012db187cabe47ac8ed136762bac6e3a: No bootstrap required, opened a new log
I20260810 00:45:35.715286 4123 raft_consensus.cc:348] T 00000000000000000000000000000000 P 012db187cabe47ac8ed136762bac6e3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012db187cabe47ac8ed136762bac6e3a" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 42593 } }
I20260810 00:45:35.715595 4123 raft_consensus.cc:374] T 00000000000000000000000000000000 P 012db187cabe47ac8ed136762bac6e3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:35.715674 4123 raft_consensus.cc:729] T 00000000000000000000000000000000 P 012db187cabe47ac8ed136762bac6e3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 012db187cabe47ac8ed136762bac6e3a, State: Initialized, Role: FOLLOWER
I20260810 00:45:35.716137 4123 consensus_queue.cc:260] T 00000000000000000000000000000000 P 012db187cabe47ac8ed136762bac6e3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012db187cabe47ac8ed136762bac6e3a" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 42593 } }
I20260810 00:45:35.716333 4123 raft_consensus.cc:388] T 00000000000000000000000000000000 P 012db187cabe47ac8ed136762bac6e3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:45:35.716447 4123 raft_consensus.cc:482] T 00000000000000000000000000000000 P 012db187cabe47ac8ed136762bac6e3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:45:35.716583 4123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 012db187cabe47ac8ed136762bac6e3a [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:35.718638 4123 raft_consensus.cc:504] T 00000000000000000000000000000000 P 012db187cabe47ac8ed136762bac6e3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012db187cabe47ac8ed136762bac6e3a" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 42593 } }
I20260810 00:45:35.719133 4123 leader_election.cc:302] T 00000000000000000000000000000000 P 012db187cabe47ac8ed136762bac6e3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 012db187cabe47ac8ed136762bac6e3a; no voters:
I20260810 00:45:35.719686 4123 leader_election.cc:288] T 00000000000000000000000000000000 P 012db187cabe47ac8ed136762bac6e3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:45:35.719839 4129 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 012db187cabe47ac8ed136762bac6e3a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:45:35.720602 4129 raft_consensus.cc:686] T 00000000000000000000000000000000 P 012db187cabe47ac8ed136762bac6e3a [term 1 LEADER]: Becoming Leader. State: Replica: 012db187cabe47ac8ed136762bac6e3a, State: Running, Role: LEADER
I20260810 00:45:35.721170 4129 consensus_queue.cc:237] T 00000000000000000000000000000000 P 012db187cabe47ac8ed136762bac6e3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012db187cabe47ac8ed136762bac6e3a" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 42593 } }
I20260810 00:45:35.721668 4123 sys_catalog.cc:564] T 00000000000000000000000000000000 P 012db187cabe47ac8ed136762bac6e3a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:45:35.724150 4130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 012db187cabe47ac8ed136762bac6e3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "012db187cabe47ac8ed136762bac6e3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012db187cabe47ac8ed136762bac6e3a" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 42593 } } }
I20260810 00:45:35.724584 4130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 012db187cabe47ac8ed136762bac6e3a [sys.catalog]: This master's current role is: LEADER
I20260810 00:45:35.724833 4132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 012db187cabe47ac8ed136762bac6e3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 012db187cabe47ac8ed136762bac6e3a. Latest consensus state: current_term: 1 leader_uuid: "012db187cabe47ac8ed136762bac6e3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "012db187cabe47ac8ed136762bac6e3a" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 42593 } } }
I20260810 00:45:35.725157 4132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 012db187cabe47ac8ed136762bac6e3a [sys.catalog]: This master's current role is: LEADER
I20260810 00:45:35.726734 4142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:45:35.731707 4142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:45:35.739279 4142 catalog_manager.cc:1347] Generated new cluster ID: e66caefa28de4124a3ebaf4a262dfc4a
I20260810 00:45:35.739391 4142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:45:35.759189 4142 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:45:35.760377 4142 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:45:35.770690 4142 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 012db187cabe47ac8ed136762bac6e3a: Generated new TSK 0
I20260810 00:45:35.771468 4142 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 00:45:35.817090 4165 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260810 00:45:35.817221 4165 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 00:45:35.818331 4127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:35.791311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33308 (local address 127.28.6.62:42593)
0810 00:45:35.791518 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:35.791528 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:35.791928 (+ 400us) server_negotiation.cc:407] Connection header received
0810 00:45:35.792220 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:35.792224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:35.792301 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:35.792450 (+ 149us) server_negotiation.cc:226] Negotiated authn=JWT
0810 00:45:35.792936 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:35.793639 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:35.795047 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:35.795516 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:35.795652 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:35.796126 (+ 474us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 00:45:35.817551 (+ 21425us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36349/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 00:45:35.817985 (+ 434us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33308: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36349/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":52}
W20260810 00:45:35.818454 4127 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33308: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36349/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260810 00:45:35.818472 4164 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:45:35.791241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:42593 (local address 127.0.0.1:33308)
0810 00:45:35.791825 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:35.791840 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:35.792000 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:35.792467 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:35.792471 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:35.792483 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0810 00:45:35.792762 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:35.792771 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:35.793793 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:35.793805 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:35.794860 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:35.794870 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:35.795642 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:35.795686 (+ 44us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:35.795846 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:35.795879 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 00:45:35.817774 (+ 21895us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36349/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 00:45:35.818040 (+ 266us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.6.62:42593: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36349/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":346,"threads_started":1}
I20260810 00:45:35.819633 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 4027
I20260810 00:45:35.820762 4115 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:45:35.913848 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 4027
2026-08-10T00:45:35Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (761 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260810 00:45:35.975697 28696 mini_oidc.cc:152] Starting JWKS server
I20260810 00:45:35.977754 28696 webserver.cc:460] Webserver started at https://127.0.0.1:39931/ using document root <none> and password file <none>
I20260810 00:45:35.978000 28696 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 00:45:35.978626 28696 webserver.cc:460] Webserver started at http://127.0.0.1:41069/ using document root <none> and password file <none>
2026-08-10T00:45:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:45:35Z Disabled control of system clock
I20260810 00:45:36.010010 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:43885
--webserver_interface=127.28.6.62
--webserver_port=0
--builtin_ntp_servers=127.28.6.20:36895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:43885
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39931/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 00:45:36.337731 4176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:36.338073 4176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:36.338243 4176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:36.345230 4176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:45:36.345389 4176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:36.345485 4176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:36.345589 4176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:45:36.345678 4176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 00:45:36.345769 4176 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 00:45:36.345851 4176 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39931/jwks.json
I20260810 00:45:36.354641 4176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:36895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:43885
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:43885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39931/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:36.355952 4176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:36.358145 4176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:36.369969 4182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:36.369998 4183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:36.369822 4185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:36.371258 4176 server_base.cc:1034] running on GCE node
I20260810 00:45:36.371959 4176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:36.373189 4176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:36.374387 4176 hybrid_clock.cc:648] HybridClock initialized: now 1786322736374330 us; error 47 us; skew 500 ppm
I20260810 00:45:36.374722 4176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 00:45:36.377194 4176 webserver.cc:460] Webserver started at http://127.28.6.62:34987/ using document root <none> and password file <none>
I20260810 00:45:36.377859 4176 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:36.377976 4176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:36.378366 4176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:36.381500 4176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "11416169ae194829878e454c5b57daf4"
format_stamp: "Formatted at 2026-08-10 00:45:36 on dist-test-slave-1nl6"
server_key: "2222d0925590b284a3a59f6e8b688b80"
server_key_iv: "57793234137bdf52fd9822d351b881a8"
server_key_version: "encryptionkey@0"
I20260810 00:45:36.382350 4176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "11416169ae194829878e454c5b57daf4"
format_stamp: "Formatted at 2026-08-10 00:45:36 on dist-test-slave-1nl6"
server_key: "2222d0925590b284a3a59f6e8b688b80"
server_key_iv: "57793234137bdf52fd9822d351b881a8"
server_key_version: "encryptionkey@0"
I20260810 00:45:36.387567 4176 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260810 00:45:36.390847 4193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:36.392202 4176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 00:45:36.392381 4176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "11416169ae194829878e454c5b57daf4"
format_stamp: "Formatted at 2026-08-10 00:45:36 on dist-test-slave-1nl6"
server_key: "2222d0925590b284a3a59f6e8b688b80"
server_key_iv: "57793234137bdf52fd9822d351b881a8"
server_key_version: "encryptionkey@0"
I20260810 00:45:36.392596 4176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:36.430477 4176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:36.431283 4176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:36.431594 4176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:36.450214 4176 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:43885
I20260810 00:45:36.450264 4265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:43885 every 8 connection(s)
I20260810 00:45:36.451565 4176 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:45:36.455343 4266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:36.459419 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 4176
I20260810 00:45:36.459645 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:45:36.460464 28696 external_mini_cluster.cc:1428] Setting key 0808fab87fba98ae898fb544a142a1aa
I20260810 00:45:36.469014 4266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11416169ae194829878e454c5b57daf4: Bootstrap starting.
I20260810 00:45:36.472844 4266 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11416169ae194829878e454c5b57daf4: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:36.473946 4266 log.cc:824] T 00000000000000000000000000000000 P 11416169ae194829878e454c5b57daf4: Log is configured to *not* fsync() on all Append() calls
I20260810 00:45:36.473953 4269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:36.462456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47644 (local address 127.28.6.62:43885)
0810 00:45:36.463494 (+ 1038us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:36.463504 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:36.463544 (+ 40us) server_negotiation.cc:407] Connection header received
0810 00:45:36.465021 (+ 1477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:36.465042 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:36.465506 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:36.465833 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:36.466493 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:36.467839 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:36.469693 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:36.470333 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:36.470472 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:36.471695 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:36.471703 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:36.471715 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:45:36.472251 (+ 536us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:36.472305 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:45:36.472325 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:36.472411 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:36.472748 (+ 337us) server_negotiation.cc:299] Negotiation successful
0810 00:45:36.473064 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":407,"threads_started":1}
I20260810 00:45:36.477064 4266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11416169ae194829878e454c5b57daf4: No bootstrap required, opened a new log
I20260810 00:45:36.477911 28696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 00:45:36.485491 4266 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11416169ae194829878e454c5b57daf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11416169ae194829878e454c5b57daf4" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 43885 } }
I20260810 00:45:36.485858 4266 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11416169ae194829878e454c5b57daf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:36.485996 4266 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11416169ae194829878e454c5b57daf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11416169ae194829878e454c5b57daf4, State: Initialized, Role: FOLLOWER
I20260810 00:45:36.486537 4266 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11416169ae194829878e454c5b57daf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11416169ae194829878e454c5b57daf4" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 43885 } }
I20260810 00:45:36.486781 4266 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11416169ae194829878e454c5b57daf4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:45:36.486960 4266 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11416169ae194829878e454c5b57daf4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:45:36.487136 4266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11416169ae194829878e454c5b57daf4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:36.490188 4266 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11416169ae194829878e454c5b57daf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11416169ae194829878e454c5b57daf4" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 43885 } }
I20260810 00:45:36.490856 4266 leader_election.cc:302] T 00000000000000000000000000000000 P 11416169ae194829878e454c5b57daf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11416169ae194829878e454c5b57daf4; no voters:
I20260810 00:45:36.491472 4266 leader_election.cc:288] T 00000000000000000000000000000000 P 11416169ae194829878e454c5b57daf4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:45:36.491597 4273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11416169ae194829878e454c5b57daf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:45:36.492377 4273 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11416169ae194829878e454c5b57daf4 [term 1 LEADER]: Becoming Leader. State: Replica: 11416169ae194829878e454c5b57daf4, State: Running, Role: LEADER
I20260810 00:45:36.492861 4273 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11416169ae194829878e454c5b57daf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11416169ae194829878e454c5b57daf4" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 43885 } }
I20260810 00:45:36.493549 4266 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11416169ae194829878e454c5b57daf4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:45:36.495772 4275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11416169ae194829878e454c5b57daf4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11416169ae194829878e454c5b57daf4. Latest consensus state: current_term: 1 leader_uuid: "11416169ae194829878e454c5b57daf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11416169ae194829878e454c5b57daf4" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 43885 } } }
I20260810 00:45:36.496117 4275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11416169ae194829878e454c5b57daf4 [sys.catalog]: This master's current role is: LEADER
I20260810 00:45:36.499137 4274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11416169ae194829878e454c5b57daf4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11416169ae194829878e454c5b57daf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11416169ae194829878e454c5b57daf4" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 43885 } } }
I20260810 00:45:36.499779 4274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11416169ae194829878e454c5b57daf4 [sys.catalog]: This master's current role is: LEADER
I20260810 00:45:36.504043 4278 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:45:36.510272 4278 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:45:36.520589 4278 catalog_manager.cc:1347] Generated new cluster ID: 801d4c9e15f4431c8df0a4475af9165b
I20260810 00:45:36.520695 4278 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:45:36.532898 4278 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:45:36.534654 4278 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:45:36.549038 4278 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11416169ae194829878e454c5b57daf4: Generated new TSK 0
I20260810 00:45:36.549817 4278 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 00:45:36.570489 4269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:36.563150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47652 (local address 127.28.6.62:43885)
0810 00:45:36.563356 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:36.563368 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:36.563758 (+ 390us) server_negotiation.cc:407] Connection header received
0810 00:45:36.563993 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:36.563997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:36.564068 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:36.564222 (+ 154us) server_negotiation.cc:226] Negotiated authn=JWT
0810 00:45:36.564706 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:36.565408 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:36.566853 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:36.567376 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:36.567520 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:36.567989 (+ 469us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 00:45:36.569870 (+ 1881us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39931/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 00:45:36.570209 (+ 339us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47652: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39931/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":62}
W20260810 00:45:36.570631 4269 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47652: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39931/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260810 00:45:36.571096 4304 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:45:36.563019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:43885 (local address 127.0.0.1:47652)
0810 00:45:36.563651 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:36.563671 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:36.563830 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:36.564243 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:36.564248 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:36.564261 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0810 00:45:36.564545 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:36.564554 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:36.565555 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:36.565561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:36.566706 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:36.566718 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:36.567511 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:36.567585 (+ 74us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:36.567728 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:36.567760 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 00:45:36.570067 (+ 2307us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39931/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 00:45:36.570318 (+ 251us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.6.62:43885: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39931/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":374,"threads_started":1}
I20260810 00:45:36.572048 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 4176
I20260810 00:45:36.573268 4260 generic_service.cc:196] Checking for leaks (request via RPC)
E20260810 00:45:36.582979 4306 webserver.cc:533] Webserver: sslize failed:
E20260810 00:45:36.583072 4306 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 00:45:36.665385 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 4176
2026-08-10T00:45:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (753 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-10T00:45:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:45:36Z Disabled control of system clock
I20260810 00:45:36.760196 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:33619
--webserver_interface=127.28.6.62
--webserver_port=0
--builtin_ntp_servers=127.28.6.20:39659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:33619
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786322687226232-28696-0/testchainca.cert with env {}
W20260810 00:45:37.092806 4313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:37.093139 4313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:37.093327 4313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:37.100453 4313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:45:37.100591 4313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:37.100693 4313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:37.100798 4313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:45:37.100888 4313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 00:45:37.100979 4313 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 00:45:37.101061 4313 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260810 00:45:37.110649 4313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:39659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:33619
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:33619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786322687226232-28696-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:37.111920 4313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:37.113260 4313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:37.125254 4319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:37.125332 4321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:37.125221 4324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:37.128297 4313 server_base.cc:1034] running on GCE node
I20260810 00:45:37.129043 4313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:37.130470 4313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:37.131752 4313 hybrid_clock.cc:648] HybridClock initialized: now 1786322737131695 us; error 51 us; skew 500 ppm
I20260810 00:45:37.132071 4313 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 00:45:37.134483 4313 webserver.cc:460] Webserver started at http://127.28.6.62:45857/ using document root <none> and password file <none>
I20260810 00:45:37.135288 4313 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:37.135392 4313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:37.135659 4313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:37.138707 4313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "fe39be1bf9f641ccbe9416245e753b73"
format_stamp: "Formatted at 2026-08-10 00:45:37 on dist-test-slave-1nl6"
server_key: "dade5ee56cc8275898a19209b6e34032"
server_key_iv: "979d413f9c0b85becbdf6105304df115"
server_key_version: "encryptionkey@0"
I20260810 00:45:37.139575 4313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "fe39be1bf9f641ccbe9416245e753b73"
format_stamp: "Formatted at 2026-08-10 00:45:37 on dist-test-slave-1nl6"
server_key: "dade5ee56cc8275898a19209b6e34032"
server_key_iv: "979d413f9c0b85becbdf6105304df115"
server_key_version: "encryptionkey@0"
I20260810 00:45:37.144526 4313 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 00:45:37.147768 4331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:37.149061 4313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 00:45:37.149255 4313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "fe39be1bf9f641ccbe9416245e753b73"
format_stamp: "Formatted at 2026-08-10 00:45:37 on dist-test-slave-1nl6"
server_key: "dade5ee56cc8275898a19209b6e34032"
server_key_iv: "979d413f9c0b85becbdf6105304df115"
server_key_version: "encryptionkey@0"
I20260810 00:45:37.149549 4313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:37.187078 4313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:37.187866 4313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:37.188179 4313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:37.206626 4313 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:33619
I20260810 00:45:37.206645 4401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:33619 every 8 connection(s)
I20260810 00:45:37.207996 4313 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:45:37.211678 4404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:37.218230 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 4313
I20260810 00:45:37.218438 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:45:37.219309 28696 external_mini_cluster.cc:1428] Setting key f0f474cf46e20d72b28bb8239cc96a18
I20260810 00:45:37.222704 4404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe39be1bf9f641ccbe9416245e753b73: Bootstrap starting.
I20260810 00:45:37.225773 4404 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe39be1bf9f641ccbe9416245e753b73: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:37.226790 4404 log.cc:824] T 00000000000000000000000000000000 P fe39be1bf9f641ccbe9416245e753b73: Log is configured to *not* fsync() on all Append() calls
I20260810 00:45:37.229543 4404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe39be1bf9f641ccbe9416245e753b73: No bootstrap required, opened a new log
I20260810 00:45:37.232851 4410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:37.221319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44992 (local address 127.28.6.62:33619)
0810 00:45:37.222289 (+ 970us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:37.222299 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:37.222339 (+ 40us) server_negotiation.cc:407] Connection header received
0810 00:45:37.224095 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:37.224117 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:37.224531 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:37.224884 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:37.225406 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:37.226758 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:37.228442 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:37.229160 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:37.229345 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:37.230586 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:37.230594 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:37.230607 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:45:37.231214 (+ 607us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:37.231293 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:45:37.231316 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:37.231462 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:37.231675 (+ 213us) server_negotiation.cc:299] Negotiation successful
0810 00:45:37.231965 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":373,"threads_started":1}
I20260810 00:45:37.236559 28696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 00:45:37.237870 4404 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe39be1bf9f641ccbe9416245e753b73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe39be1bf9f641ccbe9416245e753b73" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 33619 } }
I20260810 00:45:37.238229 4404 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe39be1bf9f641ccbe9416245e753b73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:37.238337 4404 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe39be1bf9f641ccbe9416245e753b73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe39be1bf9f641ccbe9416245e753b73, State: Initialized, Role: FOLLOWER
I20260810 00:45:37.238976 4404 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe39be1bf9f641ccbe9416245e753b73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe39be1bf9f641ccbe9416245e753b73" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 33619 } }
I20260810 00:45:37.239236 4404 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe39be1bf9f641ccbe9416245e753b73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:45:37.239390 4404 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe39be1bf9f641ccbe9416245e753b73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:45:37.239573 4404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe39be1bf9f641ccbe9416245e753b73 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:37.241673 4404 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe39be1bf9f641ccbe9416245e753b73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe39be1bf9f641ccbe9416245e753b73" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 33619 } }
I20260810 00:45:37.242229 4404 leader_election.cc:302] T 00000000000000000000000000000000 P fe39be1bf9f641ccbe9416245e753b73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe39be1bf9f641ccbe9416245e753b73; no voters:
I20260810 00:45:37.242775 4404 leader_election.cc:288] T 00000000000000000000000000000000 P fe39be1bf9f641ccbe9416245e753b73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:45:37.242870 4412 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe39be1bf9f641ccbe9416245e753b73 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:45:37.243675 4412 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe39be1bf9f641ccbe9416245e753b73 [term 1 LEADER]: Becoming Leader. State: Replica: fe39be1bf9f641ccbe9416245e753b73, State: Running, Role: LEADER
I20260810 00:45:37.244233 4412 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe39be1bf9f641ccbe9416245e753b73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe39be1bf9f641ccbe9416245e753b73" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 33619 } }
I20260810 00:45:37.244860 4404 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe39be1bf9f641ccbe9416245e753b73 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:45:37.247527 4413 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe39be1bf9f641ccbe9416245e753b73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe39be1bf9f641ccbe9416245e753b73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe39be1bf9f641ccbe9416245e753b73" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 33619 } } }
I20260810 00:45:37.247529 4415 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe39be1bf9f641ccbe9416245e753b73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe39be1bf9f641ccbe9416245e753b73. Latest consensus state: current_term: 1 leader_uuid: "fe39be1bf9f641ccbe9416245e753b73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe39be1bf9f641ccbe9416245e753b73" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 33619 } } }
I20260810 00:45:37.247975 4413 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe39be1bf9f641ccbe9416245e753b73 [sys.catalog]: This master's current role is: LEADER
I20260810 00:45:37.247982 4415 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe39be1bf9f641ccbe9416245e753b73 [sys.catalog]: This master's current role is: LEADER
I20260810 00:45:37.250056 4420 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:45:37.255291 4420 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:45:37.264504 4420 catalog_manager.cc:1347] Generated new cluster ID: 5fda67489a884bc9a86575ba6b584a78
I20260810 00:45:37.264639 4420 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:45:37.274101 4420 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:45:37.275381 4420 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:45:37.283007 4420 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe39be1bf9f641ccbe9416245e753b73: Generated new TSK 0
I20260810 00:45:37.283766 4420 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 00:45:37.338788 4410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:37.300156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45006 (local address 127.28.6.62:33619)
0810 00:45:37.300347 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:37.300355 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:37.300744 (+ 389us) server_negotiation.cc:407] Connection header received
0810 00:45:37.301010 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:37.301014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:37.301084 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:37.301248 (+ 164us) server_negotiation.cc:226] Negotiated authn=JWT
0810 00:45:37.301721 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:37.302441 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:37.303887 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:37.304322 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:37.304471 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:37.304957 (+ 486us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 00:45:37.337881 (+ 32924us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 00:45:37.338399 (+ 518us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45006: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":57}
W20260810 00:45:37.338963 4450 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:45:37.300031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:33619 (local address 127.0.0.1:45006)
0810 00:45:37.300628 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:37.300644 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:37.300808 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:37.301263 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:37.301268 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:37.301280 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0810 00:45:37.301556 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:37.301566 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:37.302596 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:37.302601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:37.303740 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:37.303751 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:37.304457 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:37.304514 (+ 57us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:37.304670 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:37.304705 (+ 35us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 00:45:37.338215 (+ 33510us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 00:45:37.338518 (+ 303us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.6.62:33619: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":329,"threads_started":1}
W20260810 00:45:37.338984 4410 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45006: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260810 00:45:37.340898 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 4313
I20260810 00:45:37.342094 4395 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:45:37.434790 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 4313
2026-08-10T00:45:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (749 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (417 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (442 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:45:38 dist-test-slave-1nl6 krb5kdc[4470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:45:38 dist-test-slave-1nl6 krb5kdc[4470](info): set up 2 sockets
Aug 10 00:45:38 dist-test-slave-1nl6 krb5kdc[4470](info): commencing operation
krb5kdc: starting...
W20260810 00:45:39.621389 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.282s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:45:39 dist-test-slave-1nl6 krb5kdc[4470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:45:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:45:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.6.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
I20260810 00:45:39.843361 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:41219
--webserver_interface=127.28.6.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:46333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:41219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:45:40.173002 4487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:40.173327 4487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:40.173496 4487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:40.180557 4487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:45:40.180697 4487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:40.180810 4487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:40.180917 4487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:45:40.181006 4487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:45:40.190361 4487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:46333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:41219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:41219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:40.191624 4487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:40.192960 4487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:40.204360 4497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:40.204404 4499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:40.204452 4501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:40.205431 4487 server_base.cc:1034] running on GCE node
I20260810 00:45:40.206143 4487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:40.207511 4487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:40.208763 4487 hybrid_clock.cc:648] HybridClock initialized: now 1786322740208704 us; error 52 us; skew 500 ppm
I20260810 00:45:40.209111 4487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:45:40 dist-test-slave-1nl6 krb5kdc[4470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322740, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:40.214879 4487 init.cc:377] Logged in from keytab as kudu/127.28.6.62@KRBTEST.COM (short username kudu)
I20260810 00:45:40.216305 4487 webserver.cc:460] Webserver started at http://127.28.6.62:35745/ using document root <none> and password file <none>
I20260810 00:45:40.216895 4487 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:40.217001 4487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:40.217267 4487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:40.220104 4487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "2ccddaffa4ca42e1ac9571e8911656ff"
format_stamp: "Formatted at 2026-08-10 00:45:40 on dist-test-slave-1nl6"
server_key: "74abfcd1771f69a6c1aa56ac57fa5a04"
server_key_iv: "e226193b708ea2f374a8c0f5d0256977"
server_key_version: "encryptionkey@0"
I20260810 00:45:40.220824 4487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "2ccddaffa4ca42e1ac9571e8911656ff"
format_stamp: "Formatted at 2026-08-10 00:45:40 on dist-test-slave-1nl6"
server_key: "74abfcd1771f69a6c1aa56ac57fa5a04"
server_key_iv: "e226193b708ea2f374a8c0f5d0256977"
server_key_version: "encryptionkey@0"
I20260810 00:45:40.225940 4487 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260810 00:45:40.229175 4508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:40.230409 4487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 00:45:40.230589 4487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "2ccddaffa4ca42e1ac9571e8911656ff"
format_stamp: "Formatted at 2026-08-10 00:45:40 on dist-test-slave-1nl6"
server_key: "74abfcd1771f69a6c1aa56ac57fa5a04"
server_key_iv: "e226193b708ea2f374a8c0f5d0256977"
server_key_version: "encryptionkey@0"
I20260810 00:45:40.230775 4487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:40.252701 4487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:40.256742 4487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:40.257027 4487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:40.275429 4487 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:41219
I20260810 00:45:40.275498 4579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:41219 every 8 connection(s)
I20260810 00:45:40.276912 4487 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:45:40.280644 4580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:40.282977 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 4487
I20260810 00:45:40.283205 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:45:40.284020 28696 external_mini_cluster.cc:1428] Setting key 5e81d6fb5d35438ceb807c867dd0702e
I20260810 00:45:40.293397 4580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ccddaffa4ca42e1ac9571e8911656ff: Bootstrap starting.
I20260810 00:45:40.297399 4580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ccddaffa4ca42e1ac9571e8911656ff: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:40.298684 4580 log.cc:824] T 00000000000000000000000000000000 P 2ccddaffa4ca42e1ac9571e8911656ff: Log is configured to *not* fsync() on all Append() calls
Aug 10 00:45:40 dist-test-slave-1nl6 krb5kdc[4470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:45:40.302465 4580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ccddaffa4ca42e1ac9571e8911656ff: No bootstrap required, opened a new log
I20260810 00:45:40.308665 4583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:40.286057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47486 (local address 127.28.6.62:41219)
0810 00:45:40.286856 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:40.286867 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:40.286958 (+ 91us) server_negotiation.cc:407] Connection header received
0810 00:45:40.288629 (+ 1671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:40.288651 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:40.289025 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:40.289349 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:40.291166 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:40.292479 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:40.294283 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:40.294981 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:40.295118 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:40.300420 (+ 5302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:40.300435 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:40.300456 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:40.300496 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:40.304328 (+ 3832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:40.304964 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:40.304976 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:40.304989 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:40.305221 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:40.305563 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:40.305568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:40.305571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:40.306226 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:40.306594 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:40.306874 (+ 280us) server_negotiation.cc:299] Negotiation successful
0810 00:45:40.307310 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":387,"threads_started":1}
I20260810 00:45:40.312588 4580 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ccddaffa4ca42e1ac9571e8911656ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ccddaffa4ca42e1ac9571e8911656ff" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 41219 } }
I20260810 00:45:40.312940 4580 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ccddaffa4ca42e1ac9571e8911656ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:40.313054 4580 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ccddaffa4ca42e1ac9571e8911656ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ccddaffa4ca42e1ac9571e8911656ff, State: Initialized, Role: FOLLOWER
I20260810 00:45:40.313650 4580 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ccddaffa4ca42e1ac9571e8911656ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ccddaffa4ca42e1ac9571e8911656ff" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 41219 } }
I20260810 00:45:40.313871 4580 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ccddaffa4ca42e1ac9571e8911656ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:45:40.313970 4580 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ccddaffa4ca42e1ac9571e8911656ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:45:40.314093 4580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ccddaffa4ca42e1ac9571e8911656ff [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:40.316233 4580 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ccddaffa4ca42e1ac9571e8911656ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ccddaffa4ca42e1ac9571e8911656ff" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 41219 } }
I20260810 00:45:40.316710 4580 leader_election.cc:302] T 00000000000000000000000000000000 P 2ccddaffa4ca42e1ac9571e8911656ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ccddaffa4ca42e1ac9571e8911656ff; no voters:
I20260810 00:45:40.317232 4580 leader_election.cc:288] T 00000000000000000000000000000000 P 2ccddaffa4ca42e1ac9571e8911656ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:45:40.317378 4587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ccddaffa4ca42e1ac9571e8911656ff [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:45:40.318025 4587 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ccddaffa4ca42e1ac9571e8911656ff [term 1 LEADER]: Becoming Leader. State: Replica: 2ccddaffa4ca42e1ac9571e8911656ff, State: Running, Role: LEADER
I20260810 00:45:40.318580 4587 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ccddaffa4ca42e1ac9571e8911656ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ccddaffa4ca42e1ac9571e8911656ff" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 41219 } }
I20260810 00:45:40.319321 4580 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ccddaffa4ca42e1ac9571e8911656ff [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:45:40.321604 4588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ccddaffa4ca42e1ac9571e8911656ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ccddaffa4ca42e1ac9571e8911656ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ccddaffa4ca42e1ac9571e8911656ff" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 41219 } } }
I20260810 00:45:40.322166 4588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ccddaffa4ca42e1ac9571e8911656ff [sys.catalog]: This master's current role is: LEADER
I20260810 00:45:40.321888 4589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ccddaffa4ca42e1ac9571e8911656ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ccddaffa4ca42e1ac9571e8911656ff. Latest consensus state: current_term: 1 leader_uuid: "2ccddaffa4ca42e1ac9571e8911656ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ccddaffa4ca42e1ac9571e8911656ff" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 41219 } } }
I20260810 00:45:40.322330 4589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ccddaffa4ca42e1ac9571e8911656ff [sys.catalog]: This master's current role is: LEADER
I20260810 00:45:40.324328 4595 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:45:40.329488 4595 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:45:40.338366 4595 catalog_manager.cc:1347] Generated new cluster ID: a3f909e77e384adb98fd12a439545411
I20260810 00:45:40.338482 4595 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:45:40.358618 4595 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:45:40.360132 4595 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:45:40.365360 4595 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ccddaffa4ca42e1ac9571e8911656ff: Generated new TSK 0
I20260810 00:45:40.366153 4595 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.6.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab.
Entry for principal kudu/127.28.6.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab.
I20260810 00:45:40.427634 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.1:0
--local_ip_for_outbound_sockets=127.28.6.1
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:41219
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:46333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:45:40.770239 4620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:40.770498 4620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:40.770601 4620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:40.777563 4620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:40.777658 4620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:40.777770 4620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.1
I20260810 00:45:40.786943 4620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:46333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:41219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.1
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:40.788070 4620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:40.789297 4620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:40.802548 4628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:40.802722 4630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:40.802796 4632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:40.803602 4620 server_base.cc:1034] running on GCE node
I20260810 00:45:40.804117 4620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:40.804956 4620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:40.806157 4620 hybrid_clock.cc:648] HybridClock initialized: now 1786322740806121 us; error 37 us; skew 500 ppm
I20260810 00:45:40.806483 4620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:45:40 dist-test-slave-1nl6 krb5kdc[4470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322740, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:40.809926 4620 init.cc:377] Logged in from keytab as kudu/127.28.6.1@KRBTEST.COM (short username kudu)
I20260810 00:45:40.811482 4620 webserver.cc:460] Webserver started at http://127.28.6.1:45585/ using document root <none> and password file <none>
I20260810 00:45:40.812110 4620 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:40.812233 4620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:40.812525 4620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:40.815570 4620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-0/data/instance:
uuid: "c492c36f2ba84304a6e558586e56837c"
format_stamp: "Formatted at 2026-08-10 00:45:40 on dist-test-slave-1nl6"
server_key: "13b1db42190dec6e9af32622ab8483fb"
server_key_iv: "606c6b30a166a4c86432d09dcbab7ad0"
server_key_version: "encryptionkey@0"
I20260810 00:45:40.816408 4620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance:
uuid: "c492c36f2ba84304a6e558586e56837c"
format_stamp: "Formatted at 2026-08-10 00:45:40 on dist-test-slave-1nl6"
server_key: "13b1db42190dec6e9af32622ab8483fb"
server_key_iv: "606c6b30a166a4c86432d09dcbab7ad0"
server_key_version: "encryptionkey@0"
I20260810 00:45:40.821761 4620 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 00:45:40.825003 4643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:40.826373 4620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 00:45:40.826552 4620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-0/wal
uuid: "c492c36f2ba84304a6e558586e56837c"
format_stamp: "Formatted at 2026-08-10 00:45:40 on dist-test-slave-1nl6"
server_key: "13b1db42190dec6e9af32622ab8483fb"
server_key_iv: "606c6b30a166a4c86432d09dcbab7ad0"
server_key_version: "encryptionkey@0"
I20260810 00:45:40.826750 4620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:40.846592 4620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:40.850945 4620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:40.851243 4620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:40.852176 4620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:45:40.853475 4620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:45:40.853586 4620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:40.853739 4620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:45:40.853818 4620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:40.888670 4620 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.1:45401
I20260810 00:45:40.888726 4805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.1:45401 every 8 connection(s)
I20260810 00:45:40.889978 4620 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
I20260810 00:45:40.891427 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 4620
I20260810 00:45:40.891644 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance
I20260810 00:45:40.892518 28696 external_mini_cluster.cc:1428] Setting key 399bf1683327c644b0d90c0881aea9d1
Aug 10 00:45:40 dist-test-slave-1nl6 krb5kdc[4470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322740, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:45:40.911085 4810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:40.892942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:53531 (local address 127.28.6.62:41219)
0810 00:45:40.893447 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:40.893452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:40.894629 (+ 1177us) server_negotiation.cc:407] Connection header received
0810 00:45:40.896447 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:40.896453 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:40.896541 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:40.896660 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:40.898207 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:40.899036 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:40.900550 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:40.900946 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:40.901084 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:40.904700 (+ 3616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:40.904710 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:40.904717 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:40.904760 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:40.907414 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:40.908386 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:40.908390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:40.908392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:40.908504 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:40.909036 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:40.909039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:40.909041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:40.909519 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:40.909650 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:40.910581 (+ 931us) server_negotiation.cc:299] Negotiation successful
0810 00:45:40.910762 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":241,"threads_started":1}
I20260810 00:45:40.912554 4811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:40.893548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:41219 (local address 127.28.6.1:53531)
0810 00:45:40.894449 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:40.894502 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:40.896190 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:40.896921 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:40.896929 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:40.897476 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:40.898033 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:40.898050 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:40.899201 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:40.899206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:40.900387 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:40.900400 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:40.901174 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:40.901183 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:40.901420 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:40.902069 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:40.902095 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:40.904470 (+ 2375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:40.907566 (+ 3096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:40.907578 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:40.907596 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:40.908259 (+ 663us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:40.908610 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:40.908618 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:40.908625 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:40.908923 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:40.909685 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:40.909697 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:40.910159 (+ 462us) client_negotiation.cc:769] Sending connection context
0810 00:45:40.910473 (+ 314us) client_negotiation.cc:240] Negotiation successful
0810 00:45:40.911118 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":335,"threads_started":1}
I20260810 00:45:40.914233 4806 heartbeater.cc:347] Connected to a master server at 127.28.6.62:41219
I20260810 00:45:40.914549 4806 heartbeater.cc:464] Registering TS with master...
I20260810 00:45:40.915340 4806 heartbeater.cc:511] Master 127.28.6.62:41219 requested a full tablet report, sending...
I20260810 00:45:40.917698 4535 ts_manager.cc:194] Registered new tserver with Master: c492c36f2ba84304a6e558586e56837c (127.28.6.1:45401)
WARNING: no policy specified for kudu/127.28.6.2@KRBTEST.COM; defaulting to no policy
I20260810 00:45:40.919907 4535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.1@KRBTEST.COM'} at 127.28.6.1:53531
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab.
Entry for principal kudu/127.28.6.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab.
I20260810 00:45:40.947517 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.2:0
--local_ip_for_outbound_sockets=127.28.6.2
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:41219
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:46333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:45:41.280618 4816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:41.280861 4816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:41.281025 4816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:41.288050 4816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:41.288143 4816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:41.288360 4816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.2
I20260810 00:45:41.297662 4816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:46333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:41219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.2
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:41.298826 4816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:41.300122 4816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:41.313012 4831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:41.313091 4830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:41.313087 4833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:41.314038 4816 server_base.cc:1034] running on GCE node
I20260810 00:45:41.314577 4816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:41.315469 4816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:41.316664 4816 hybrid_clock.cc:648] HybridClock initialized: now 1786322741316624 us; error 37 us; skew 500 ppm
I20260810 00:45:41.316975 4816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:45:41 dist-test-slave-1nl6 krb5kdc[4470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322741, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:41.320210 4816 init.cc:377] Logged in from keytab as kudu/127.28.6.2@KRBTEST.COM (short username kudu)
I20260810 00:45:41.321664 4816 webserver.cc:460] Webserver started at http://127.28.6.2:43023/ using document root <none> and password file <none>
I20260810 00:45:41.322306 4816 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:41.322458 4816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:41.322781 4816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:41.325785 4816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-1/data/instance:
uuid: "668c257f6aab4275885b91d1b60d8cbb"
format_stamp: "Formatted at 2026-08-10 00:45:41 on dist-test-slave-1nl6"
server_key: "6134bfb5e2f50d322fc801830cda86a4"
server_key_iv: "71131967a7439c19d53c61e7a79f12b6"
server_key_version: "encryptionkey@0"
I20260810 00:45:41.326736 4816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance:
uuid: "668c257f6aab4275885b91d1b60d8cbb"
format_stamp: "Formatted at 2026-08-10 00:45:41 on dist-test-slave-1nl6"
server_key: "6134bfb5e2f50d322fc801830cda86a4"
server_key_iv: "71131967a7439c19d53c61e7a79f12b6"
server_key_version: "encryptionkey@0"
I20260810 00:45:41.332407 4816 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 00:45:41.335937 4845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:41.337230 4816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 00:45:41.337420 4816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-1/wal
uuid: "668c257f6aab4275885b91d1b60d8cbb"
format_stamp: "Formatted at 2026-08-10 00:45:41 on dist-test-slave-1nl6"
server_key: "6134bfb5e2f50d322fc801830cda86a4"
server_key_iv: "71131967a7439c19d53c61e7a79f12b6"
server_key_version: "encryptionkey@0"
I20260810 00:45:41.337628 4816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:41.352116 4816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:41.356361 4816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:41.356658 4816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:41.357607 4816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:45:41.358891 4816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:45:41.359042 4816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:41.359194 4816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:45:41.359273 4816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:41.394142 4816 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.2:40329
I20260810 00:45:41.394229 5006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.2:40329 every 8 connection(s)
I20260810 00:45:41.395454 4816 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
I20260810 00:45:41.397737 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 4816
I20260810 00:45:41.397914 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance
I20260810 00:45:41.398463 28696 external_mini_cluster.cc:1428] Setting key 4b1e959fc8df271805e22ba926f0ac8e
Aug 10 00:45:41 dist-test-slave-1nl6 krb5kdc[4470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322741, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:45:41.417415 4810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:41.397987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:51321 (local address 127.28.6.62:41219)
0810 00:45:41.398232 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:41.398239 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:41.399482 (+ 1243us) server_negotiation.cc:407] Connection header received
0810 00:45:41.401593 (+ 2111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:41.401601 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:41.401692 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:41.401807 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:41.403304 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:41.404429 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:41.405969 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:41.406730 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:41.406873 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:41.410534 (+ 3661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:41.410539 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:41.410542 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:41.410573 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:41.414039 (+ 3466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:41.414775 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:41.414783 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:41.414790 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:41.414980 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:41.415431 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:41.415438 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:41.415444 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:41.415837 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:41.416009 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:41.416682 (+ 673us) server_negotiation.cc:299] Negotiation successful
0810 00:45:41.416948 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 00:45:41.417972 5009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:41.398428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:41219 (local address 127.28.6.2:51321)
0810 00:45:41.399310 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:41.399364 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:41.401332 (+ 1968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:41.402000 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:41.402008 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:41.402492 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:41.403136 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:41.403156 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:41.404577 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:41.404580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:41.405806 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:41.405818 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:41.406986 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:41.406992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:41.407201 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:41.407882 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:41.407905 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:41.410307 (+ 2402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:41.414200 (+ 3893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:41.414207 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:41.414226 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:41.414631 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:41.415081 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:41.415084 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:41.415086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:41.415296 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:41.415973 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:41.415979 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:41.416297 (+ 318us) client_negotiation.cc:769] Sending connection context
0810 00:45:41.416517 (+ 220us) client_negotiation.cc:240] Negotiation successful
0810 00:45:41.416882 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":409,"threads_started":1}
I20260810 00:45:41.419541 5007 heartbeater.cc:347] Connected to a master server at 127.28.6.62:41219
I20260810 00:45:41.419885 5007 heartbeater.cc:464] Registering TS with master...
I20260810 00:45:41.420706 5007 heartbeater.cc:511] Master 127.28.6.62:41219 requested a full tablet report, sending...
I20260810 00:45:41.422224 4535 ts_manager.cc:194] Registered new tserver with Master: 668c257f6aab4275885b91d1b60d8cbb (127.28.6.2:40329)
I20260810 00:45:41.423095 4535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.2@KRBTEST.COM'} at 127.28.6.2:51321
WARNING: no policy specified for kudu/127.28.6.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab.
Entry for principal kudu/127.28.6.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab.
I20260810 00:45:41.451650 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.3:0
--local_ip_for_outbound_sockets=127.28.6.3
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:41219
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:46333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:45:41.784196 5012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:41.784453 5012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:41.784605 5012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:41.792245 5012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:41.792362 5012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:41.792534 5012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.3
I20260810 00:45:41.802172 5012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:46333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:41219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.3
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:41.803412 5012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:41.804648 5012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:41.817349 5022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:41.817409 5023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:41.817436 5025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:41.818570 5012 server_base.cc:1034] running on GCE node
I20260810 00:45:41.819257 5012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:41.820194 5012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:41.821414 5012 hybrid_clock.cc:648] HybridClock initialized: now 1786322741821383 us; error 42 us; skew 500 ppm
I20260810 00:45:41.821750 5012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:45:41 dist-test-slave-1nl6 krb5kdc[4470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322741, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:41.825106 5012 init.cc:377] Logged in from keytab as kudu/127.28.6.3@KRBTEST.COM (short username kudu)
I20260810 00:45:41.826525 5012 webserver.cc:460] Webserver started at http://127.28.6.3:44075/ using document root <none> and password file <none>
I20260810 00:45:41.827247 5012 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:41.827399 5012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:41.827695 5012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:41.830605 5012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-2/data/instance:
uuid: "ceab039ab04449cdbef855137afd9e80"
format_stamp: "Formatted at 2026-08-10 00:45:41 on dist-test-slave-1nl6"
server_key: "9fdb7fe1e275d5b5663c129db2a9f387"
server_key_iv: "7ad6f482c4dc46541f7497c159371f0c"
server_key_version: "encryptionkey@0"
I20260810 00:45:41.831529 5012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance:
uuid: "ceab039ab04449cdbef855137afd9e80"
format_stamp: "Formatted at 2026-08-10 00:45:41 on dist-test-slave-1nl6"
server_key: "9fdb7fe1e275d5b5663c129db2a9f387"
server_key_iv: "7ad6f482c4dc46541f7497c159371f0c"
server_key_version: "encryptionkey@0"
I20260810 00:45:41.836828 5012 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 00:45:41.840124 5034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:41.841480 5012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260810 00:45:41.841648 5012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-2/wal
uuid: "ceab039ab04449cdbef855137afd9e80"
format_stamp: "Formatted at 2026-08-10 00:45:41 on dist-test-slave-1nl6"
server_key: "9fdb7fe1e275d5b5663c129db2a9f387"
server_key_iv: "7ad6f482c4dc46541f7497c159371f0c"
server_key_version: "encryptionkey@0"
I20260810 00:45:41.841853 5012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:41.859995 5012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:41.863754 5012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:41.864051 5012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:41.864966 5012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:45:41.866251 5012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:45:41.866361 5012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:41.866518 5012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:45:41.866592 5012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:41.903327 5012 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.3:34631
I20260810 00:45:41.903376 5186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.3:34631 every 8 connection(s)
I20260810 00:45:41.904578 5012 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
I20260810 00:45:41.911285 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 5012
I20260810 00:45:41.911448 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance
I20260810 00:45:41.911904 28696 external_mini_cluster.cc:1428] Setting key b5f155cbc85fff9f4c1638b79883d9ad
Aug 10 00:45:41 dist-test-slave-1nl6 krb5kdc[4470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322741, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:45:41.922870 4806 heartbeater.cc:503] Master 127.28.6.62:41219 was elected leader, sending a full tablet report...
I20260810 00:45:41.924211 4810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:41.907003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:58549 (local address 127.28.6.62:41219)
0810 00:45:41.907189 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:41.907193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:41.908431 (+ 1238us) server_negotiation.cc:407] Connection header received
0810 00:45:41.910206 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:41.910212 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:41.910299 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:41.910425 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:41.912072 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:41.913076 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:41.914600 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:41.915068 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:41.915203 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:41.918415 (+ 3212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:41.918421 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:41.918426 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:41.918468 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:41.921086 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:41.921782 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:41.921787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:41.921791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:41.921982 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:41.922413 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:41.922418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:41.922423 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:41.922676 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:41.922838 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:41.923628 (+ 790us) server_negotiation.cc:299] Negotiation successful
0810 00:45:41.923859 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 00:45:41.925100 5193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:41.907419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:41219 (local address 127.28.6.3:58549)
0810 00:45:41.908250 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:41.908296 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:41.909939 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:41.910657 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:41.910664 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:41.911241 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:41.911866 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:41.911883 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:41.913243 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:41.913248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:41.914430 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:41.914442 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:41.915330 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:41.915339 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:41.915557 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:41.916133 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:41.916155 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:41.918207 (+ 2052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:41.921229 (+ 3022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:41.921237 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:41.921251 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:41.921650 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:41.922100 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:41.922105 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:41.922109 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:41.922287 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:41.922817 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:41.922825 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:41.923236 (+ 411us) client_negotiation.cc:769] Sending connection context
0810 00:45:41.923506 (+ 270us) client_negotiation.cc:240] Negotiation successful
0810 00:45:41.923982 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":303,"threads_started":1}
I20260810 00:45:41.926577 5191 heartbeater.cc:347] Connected to a master server at 127.28.6.62:41219
I20260810 00:45:41.926966 5191 heartbeater.cc:464] Registering TS with master...
I20260810 00:45:41.927805 5191 heartbeater.cc:511] Master 127.28.6.62:41219 requested a full tablet report, sending...
I20260810 00:45:41.929207 4533 ts_manager.cc:194] Registered new tserver with Master: ceab039ab04449cdbef855137afd9e80 (127.28.6.3:34631)
I20260810 00:45:41.930025 4533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.3@KRBTEST.COM'} at 127.28.6.3:58549
I20260810 00:45:41.939440 28696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/krb5kdc/test-admin.keytab.
Aug 10 00:45:41 dist-test-slave-1nl6 krb5kdc[4470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:41.969169 28696 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260810 00:45:41.974315 5199 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:45:41.971922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40198 (local address 127.28.6.1:45401)
0810 00:45:41.972570 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:41.972577 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:41.972604 (+ 27us) server_negotiation.cc:407] Connection header received
0810 00:45:41.972667 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:41.972674 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:41.972894 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:41.973037 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:41.973854 (+ 817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40198: BlockingRecv error: recv got EOF from 127.0.0.1:40198 (error 108)
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":346,"threads_started":1}
W20260810 00:45:41.974655 28696 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/krb5kdc/krb5cc' of type FILE
Aug 10 00:45:41 dist-test-slave-1nl6 krb5kdc[4470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:41.976248 28696 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 00:45:41 dist-test-slave-1nl6 krb5kdc[4470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:41.977620 28696 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 00:45:41 dist-test-slave-1nl6 krb5kdc[4470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.1@KRBTEST.COM
I20260810 00:45:41.992611 5199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:41.979129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40206 (local address 127.28.6.1:45401)
0810 00:45:41.979336 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:41.979341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:41.979579 (+ 238us) server_negotiation.cc:407] Connection header received
0810 00:45:41.979751 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:41.979756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:41.979821 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:41.979920 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:41.980908 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:41.981890 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:41.983213 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:41.983758 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:41.983899 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:41.987334 (+ 3435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:41.987345 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:41.987360 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:41.987389 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:41.989858 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:41.990504 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:41.990511 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:41.990513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:41.990631 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:41.991084 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:41.991087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:41.991089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:41.991417 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:41.991725 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:41.992067 (+ 342us) server_negotiation.cc:299] Negotiation successful
0810 00:45:41.992345 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260810 00:45:41.998193 5199 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:45:41.996512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40222 (local address 127.28.6.1:45401)
0810 00:45:41.996701 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:41.996708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:41.996863 (+ 155us) server_negotiation.cc:407] Connection header received
0810 00:45:41.996970 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:41.996973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:41.997021 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:41.997103 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:41.997984 (+ 881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40222: BlockingRecv error: recv got EOF from 127.0.0.1:40222 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260810 00:45:41.998458 28696 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/krb5kdc/krb5cc' of type FILE
Aug 10 00:45:41 dist-test-slave-1nl6 krb5kdc[4470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:41.999984 28696 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 00:45:41 dist-test-slave-1nl6 krb5kdc[4470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:42.001225 28696 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 00:45:42 dist-test-slave-1nl6 krb5kdc[4470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.1@KRBTEST.COM
I20260810 00:45:42.014516 5199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:42.002392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40234 (local address 127.28.6.1:45401)
0810 00:45:42.002547 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:42.002551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:42.002952 (+ 401us) server_negotiation.cc:407] Connection header received
0810 00:45:42.003095 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:42.003098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:42.003147 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:42.003223 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:42.004099 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:42.004749 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:42.005853 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:42.006272 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:42.006373 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:42.009704 (+ 3331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:42.009708 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:42.009712 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:42.009754 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:42.012310 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:42.012955 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:42.012959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:42.012961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:42.013080 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:42.013390 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:42.013393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:42.013395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:42.013640 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:42.013842 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:42.014053 (+ 211us) server_negotiation.cc:299] Negotiation successful
0810 00:45:42.014242 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260810 00:45:42.018965 5199 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:45:42.017116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40236 (local address 127.28.6.1:45401)
0810 00:45:42.017321 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:42.017329 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:42.017616 (+ 287us) server_negotiation.cc:407] Connection header received
0810 00:45:42.017788 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:42.017800 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:42.017865 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:42.017965 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:42.018723 (+ 758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40236: BlockingRecv error: recv got EOF from 127.0.0.1:40236 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260810 00:45:42.019135 28696 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786322687226232-28696-0/krb5kdc/krb5cc' of type FILE
Aug 10 00:45:42 dist-test-slave-1nl6 krb5kdc[4470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:42.020803 28696 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 00:45:42 dist-test-slave-1nl6 krb5kdc[4470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:42.022146 28696 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 00:45:42 dist-test-slave-1nl6 krb5kdc[4470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.1@KRBTEST.COM
I20260810 00:45:42.036867 5199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:42.023798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40246 (local address 127.28.6.1:45401)
0810 00:45:42.024014 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:42.024019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:42.024248 (+ 229us) server_negotiation.cc:407] Connection header received
0810 00:45:42.024343 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:42.024346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:42.024394 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:42.024467 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:42.025417 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:42.026168 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:42.027565 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:42.027978 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:42.028075 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:42.031719 (+ 3644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:42.031724 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:42.031727 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:42.031756 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:42.034132 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:42.034816 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:42.034822 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:42.034828 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:42.035024 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:42.035449 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:42.035454 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:42.035458 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:42.035758 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:42.036082 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:42.036358 (+ 276us) server_negotiation.cc:299] Negotiation successful
0810 00:45:42.036561 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 00:45:42.038262 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 4620
I20260810 00:45:42.049369 5199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:42.038988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40252 (local address 127.28.6.1:45401)
0810 00:45:42.039165 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:42.039173 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:42.039425 (+ 252us) server_negotiation.cc:407] Connection header received
0810 00:45:42.039550 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:42.039553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:42.039603 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:42.039677 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:42.040507 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:42.041214 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:42.042373 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:42.042773 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:42.042869 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:42.044655 (+ 1786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:42.044659 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:42.044663 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:42.044690 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:42.047288 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:42.047814 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:42.047817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:42.047819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:42.047927 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:42.048257 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:42.048260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:42.048262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:42.048487 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:42.048664 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:42.048869 (+ 205us) server_negotiation.cc:299] Negotiation successful
0810 00:45:42.049053 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 00:45:42.049917 4801 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:45:42.131693 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 4620
I20260810 00:45:42.159584 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 4816
Aug 10 00:45:42 dist-test-slave-1nl6 krb5kdc[4470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.2@KRBTEST.COM
I20260810 00:45:42.174563 5228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:42.160498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58488 (local address 127.28.6.2:40329)
0810 00:45:42.161069 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:42.161076 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:42.161103 (+ 27us) server_negotiation.cc:407] Connection header received
0810 00:45:42.161167 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:42.161174 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:42.161390 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:42.161518 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:42.162460 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:42.163549 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:42.164726 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:42.165244 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:42.165348 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:42.169133 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:42.169144 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:42.169159 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:42.169190 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:42.172223 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:42.172802 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:42.172809 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:42.172811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:42.172929 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:42.173284 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:42.173288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:42.173290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:42.173612 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:42.173842 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:42.174066 (+ 224us) server_negotiation.cc:299] Negotiation successful
0810 00:45:42.174267 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":318,"threads_started":1}
I20260810 00:45:42.175843 5002 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:45:42.258261 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 4816
I20260810 00:45:42.284358 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 5012
Aug 10 00:45:42 dist-test-slave-1nl6 krb5kdc[4470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.3@KRBTEST.COM
I20260810 00:45:42.300581 5231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:42.285386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50526 (local address 127.28.6.3:34631)
0810 00:45:42.286040 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:42.286047 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:42.286078 (+ 31us) server_negotiation.cc:407] Connection header received
0810 00:45:42.286140 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:42.286145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:42.286356 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:42.286536 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:42.287738 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:42.288744 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:42.290049 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:42.290647 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:42.290799 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:42.294999 (+ 4200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:42.295011 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:42.295026 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:42.295058 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:42.297711 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:42.298421 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:42.298429 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:42.298431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:42.298559 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:42.298974 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:42.298983 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:42.298987 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:42.299352 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:42.299679 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:42.300007 (+ 328us) server_negotiation.cc:299] Negotiation successful
0810 00:45:42.300283 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":345,"threads_started":1}
I20260810 00:45:42.301910 5182 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:45:42.384311 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 5012
I20260810 00:45:42.410796 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 4487
I20260810 00:45:42.412072 4573 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:45:42.517004 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 4487
2026-08-10T00:45:42Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4228 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:45:42 dist-test-slave-1nl6 krb5kdc[5237](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:45:42 dist-test-slave-1nl6 krb5kdc[5237](info): set up 2 sockets
Aug 10 00:45:42 dist-test-slave-1nl6 krb5kdc[5237](info): commencing operation
krb5kdc: starting...
W20260810 00:45:43.802567 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.235s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:45:43 dist-test-slave-1nl6 krb5kdc[5237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:45:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:45:43Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.6.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.6.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.6.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.62.keytab.
Entry for principal oryx/127.28.6.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.62.keytab.
I20260810 00:45:44.046555 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:39675
--webserver_interface=127.28.6.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.62.keytab
--principal=oryx/127.28.6.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:37683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:39675
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:45:44.374498 5253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:44.374787 5253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:44.374878 5253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:44.382639 5253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:45:44.382747 5253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:44.382805 5253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:44.382859 5253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:45:44.382939 5253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:45:44.392686 5253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:37683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:39675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.62.keytab
--principal=oryx/127.28.6.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:39675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:44.393985 5253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:44.395363 5253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:44.407027 5259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:44.407090 5262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:44.407055 5260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:44.408174 5253 server_base.cc:1034] running on GCE node
I20260810 00:45:44.408898 5253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:44.410204 5253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:44.411429 5253 hybrid_clock.cc:648] HybridClock initialized: now 1786322744411390 us; error 45 us; skew 500 ppm
I20260810 00:45:44.411753 5253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:45:44 dist-test-slave-1nl6 krb5kdc[5237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322744, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.6.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:44.415004 5253 init.cc:377] Logged in from keytab as oryx/127.28.6.62@KRBTEST.COM (short username oryx)
I20260810 00:45:44.416508 5253 webserver.cc:460] Webserver started at http://127.28.6.62:36041/ using document root <none> and password file <none>
I20260810 00:45:44.417171 5253 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:44.417325 5253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:44.417627 5253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:44.420550 5253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "e6e4f5d8a3384fbcbee45036aad474fa"
format_stamp: "Formatted at 2026-08-10 00:45:44 on dist-test-slave-1nl6"
server_key: "a68e04250714ee87ebc549d6fb51eb45"
server_key_iv: "e64d59fc13839dfa25f88ccf6248b69f"
server_key_version: "encryptionkey@0"
I20260810 00:45:44.421322 5253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "e6e4f5d8a3384fbcbee45036aad474fa"
format_stamp: "Formatted at 2026-08-10 00:45:44 on dist-test-slave-1nl6"
server_key: "a68e04250714ee87ebc549d6fb51eb45"
server_key_iv: "e64d59fc13839dfa25f88ccf6248b69f"
server_key_version: "encryptionkey@0"
I20260810 00:45:44.426328 5253 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 00:45:44.429575 5269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:44.430773 5253 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 00:45:44.431001 5253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "e6e4f5d8a3384fbcbee45036aad474fa"
format_stamp: "Formatted at 2026-08-10 00:45:44 on dist-test-slave-1nl6"
server_key: "a68e04250714ee87ebc549d6fb51eb45"
server_key_iv: "e64d59fc13839dfa25f88ccf6248b69f"
server_key_version: "encryptionkey@0"
I20260810 00:45:44.431188 5253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:44.450641 5253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:44.455124 5253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:44.455402 5253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:44.473758 5253 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:39675
I20260810 00:45:44.473845 5341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:39675 every 8 connection(s)
I20260810 00:45:44.475101 5253 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:45:44.477051 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 5253
I20260810 00:45:44.477257 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:45:44.477932 28696 external_mini_cluster.cc:1428] Setting key 8ca42e0f2d3ec4adc1ef63fcd17bc16f
I20260810 00:45:44.479022 5343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 00:45:44 dist-test-slave-1nl6 krb5kdc[5237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM
I20260810 00:45:44.493244 5343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa: Bootstrap starting.
I20260810 00:45:44.496536 5343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:44.497581 5343 log.cc:824] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa: Log is configured to *not* fsync() on all Append() calls
I20260810 00:45:44.500511 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:44.479933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53588 (local address 127.28.6.62:39675)
0810 00:45:44.480780 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:44.480790 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:44.480829 (+ 39us) server_negotiation.cc:407] Connection header received
0810 00:45:44.482313 (+ 1484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:44.482341 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:44.482692 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:44.483107 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:44.484927 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:44.486233 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:44.488022 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:44.488650 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:44.488771 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:44.493331 (+ 4560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:44.493338 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:44.493351 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:44.493429 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:44.496332 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:44.497079 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:44.497086 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:44.497091 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:44.497251 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:44.497653 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:44.497657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:44.497660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:44.498240 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:44.498617 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:44.499014 (+ 397us) server_negotiation.cc:299] Negotiation successful
0810 00:45:44.499458 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":382,"threads_started":1}
I20260810 00:45:44.501008 5343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa: No bootstrap required, opened a new log
I20260810 00:45:44.508752 5343 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } }
I20260810 00:45:44.509198 5343 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:44.509359 5343 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6e4f5d8a3384fbcbee45036aad474fa, State: Initialized, Role: FOLLOWER
I20260810 00:45:44.510030 5343 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } }
I20260810 00:45:44.510257 5343 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:45:44.510362 5343 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:45:44.510486 5343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:44.512563 5343 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } }
I20260810 00:45:44.513058 5343 leader_election.cc:302] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6e4f5d8a3384fbcbee45036aad474fa; no voters:
I20260810 00:45:44.513653 5343 leader_election.cc:288] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:45:44.513799 5351 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:45:44.515627 5351 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [term 1 LEADER]: Becoming Leader. State: Replica: e6e4f5d8a3384fbcbee45036aad474fa, State: Running, Role: LEADER
I20260810 00:45:44.515903 5343 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:45:44.516211 5351 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } }
I20260810 00:45:44.519801 5355 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6e4f5d8a3384fbcbee45036aad474fa. Latest consensus state: current_term: 1 leader_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } }
I20260810 00:45:44.520150 5355 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [sys.catalog]: This master's current role is: LEADER
I20260810 00:45:44.519198 5353 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } }
I20260810 00:45:44.520718 5353 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [sys.catalog]: This master's current role is: LEADER
I20260810 00:45:44.526844 5363 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:45:44.531744 5363 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:45:44.540508 5363 catalog_manager.cc:1347] Generated new cluster ID: 55a845b735744815b615e8a0736d9f90
I20260810 00:45:44.540627 5363 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:45:44.557036 5363 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:45:44.558808 5363 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:45:44.573984 5363 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa: Generated new TSK 0
I20260810 00:45:44.574877 5363 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.6.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.6.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.6.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.1.keytab.
Entry for principal oryx/127.28.6.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.1.keytab.
I20260810 00:45:44.642164 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.1:0
--local_ip_for_outbound_sockets=127.28.6.1
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:39675
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.1.keytab
--principal=oryx/127.28.6.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:37683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:45:44.972524 5384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:44.972811 5384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:44.972908 5384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:44.980612 5384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:44.980719 5384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:44.980839 5384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.1
I20260810 00:45:44.990339 5384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:37683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.1.keytab
--principal=oryx/127.28.6.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:39675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.1
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:44.991508 5384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:44.992839 5384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:45.005739 5395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:45.005785 5393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:45.005716 5391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:45.006868 5384 server_base.cc:1034] running on GCE node
I20260810 00:45:45.007521 5384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:45.008419 5384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:45.009629 5384 hybrid_clock.cc:648] HybridClock initialized: now 1786322745009567 us; error 51 us; skew 500 ppm
I20260810 00:45:45.009971 5384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:45:45 dist-test-slave-1nl6 krb5kdc[5237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322745, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.6.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:45.013173 5384 init.cc:377] Logged in from keytab as oryx/127.28.6.1@KRBTEST.COM (short username oryx)
I20260810 00:45:45.014617 5384 webserver.cc:460] Webserver started at http://127.28.6.1:40279/ using document root <none> and password file <none>
I20260810 00:45:45.015359 5384 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:45.015494 5384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:45.015831 5384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:45.018980 5384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-0/data/instance:
uuid: "6668f2f5a228466b8d3b7b8de4661e3d"
format_stamp: "Formatted at 2026-08-10 00:45:45 on dist-test-slave-1nl6"
server_key: "bfede56bf2b8249df9cdf93886b2fc75"
server_key_iv: "e83bb66301fec968f3d6c6293b6a61df"
server_key_version: "encryptionkey@0"
I20260810 00:45:45.019773 5384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance:
uuid: "6668f2f5a228466b8d3b7b8de4661e3d"
format_stamp: "Formatted at 2026-08-10 00:45:45 on dist-test-slave-1nl6"
server_key: "bfede56bf2b8249df9cdf93886b2fc75"
server_key_iv: "e83bb66301fec968f3d6c6293b6a61df"
server_key_version: "encryptionkey@0"
I20260810 00:45:45.024830 5384 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 00:45:45.028174 5407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:45.029541 5384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 00:45:45.029713 5384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-0/wal
uuid: "6668f2f5a228466b8d3b7b8de4661e3d"
format_stamp: "Formatted at 2026-08-10 00:45:45 on dist-test-slave-1nl6"
server_key: "bfede56bf2b8249df9cdf93886b2fc75"
server_key_iv: "e83bb66301fec968f3d6c6293b6a61df"
server_key_version: "encryptionkey@0"
I20260810 00:45:45.029912 5384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:45.044736 5384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:45.048727 5384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:45.049017 5384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:45.049998 5384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:45:45.051342 5384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:45:45.051414 5384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:45.051571 5384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:45:45.051644 5384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:45.086712 5384 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.1:42679
I20260810 00:45:45.086784 5576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.1:42679 every 8 connection(s)
I20260810 00:45:45.088038 5384 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
I20260810 00:45:45.093472 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 5384
I20260810 00:45:45.093683 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance
I20260810 00:45:45.094532 28696 external_mini_cluster.cc:1428] Setting key 95c7cf41d8920eb7d3e7d312ac98d65f
Aug 10 00:45:45 dist-test-slave-1nl6 krb5kdc[5237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322745, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.6.1@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM
I20260810 00:45:45.107559 5584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:45.090656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:38971 (local address 127.28.6.62:39675)
0810 00:45:45.091125 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:45.091132 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:45.092175 (+ 1043us) server_negotiation.cc:407] Connection header received
0810 00:45:45.094030 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:45.094036 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:45.094125 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:45.094278 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:45.095915 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:45.096614 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:45.098132 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:45.098506 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:45.098642 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:45.102001 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:45.102006 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:45.102010 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:45.102039 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:45.104596 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:45.105289 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:45.105294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:45.105296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:45.105415 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:45.105855 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:45.105859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:45.105861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:45.106101 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:45.106270 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:45.106977 (+ 707us) server_negotiation.cc:299] Negotiation successful
0810 00:45:45.107202 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":333,"thread_start_us":199,"threads_started":1}
I20260810 00:45:45.108433 5585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:45.091288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.28.6.1:38971)
0810 00:45:45.092004 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:45.092042 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:45.093716 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:45.094476 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:45.094484 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:45.095070 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:45.095745 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:45.095765 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:45.096786 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:45.096791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:45.097966 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:45.097979 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:45.098726 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:45.098735 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:45.099023 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:45.099689 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:45.099710 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:45.101788 (+ 2078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:45.104732 (+ 2944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:45.104741 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:45.104756 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:45.105152 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:45.105528 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:45.105532 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:45.105536 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:45.105719 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:45.106233 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:45.106240 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:45.106561 (+ 321us) client_negotiation.cc:769] Sending connection context
0810 00:45:45.106804 (+ 243us) client_negotiation.cc:240] Negotiation successful
0810 00:45:45.107305 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":272,"threads_started":1}
I20260810 00:45:45.109984 5579 heartbeater.cc:347] Connected to a master server at 127.28.6.62:39675
I20260810 00:45:45.110268 5579 heartbeater.cc:464] Registering TS with master...
I20260810 00:45:45.111037 5579 heartbeater.cc:511] Master 127.28.6.62:39675 requested a full tablet report, sending...
I20260810 00:45:45.113299 5292 ts_manager.cc:194] Registered new tserver with Master: 6668f2f5a228466b8d3b7b8de4661e3d (127.28.6.1:42679)
I20260810 00:45:45.115610 5292 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.6.1@KRBTEST.COM'} at 127.28.6.1:38971
WARNING: no policy specified for oryx/127.28.6.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.6.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.6.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.2.keytab.
Entry for principal oryx/127.28.6.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.2.keytab.
I20260810 00:45:45.145790 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.2:0
--local_ip_for_outbound_sockets=127.28.6.2
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:39675
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.2.keytab
--principal=oryx/127.28.6.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:37683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:45:45.478060 5588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:45.478402 5588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:45.478572 5588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:45.489787 5588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:45.489912 5588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:45.490079 5588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.2
I20260810 00:45:45.499933 5588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:37683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.2.keytab
--principal=oryx/127.28.6.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:39675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.2
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:45.501050 5588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:45.502295 5588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:45.515275 5599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:45.515316 5596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:45.515432 5595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:45.520155 5588 server_base.cc:1034] running on GCE node
I20260810 00:45:45.520747 5588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:45.521622 5588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:45.522851 5588 hybrid_clock.cc:648] HybridClock initialized: now 1786322745522799 us; error 40 us; skew 500 ppm
I20260810 00:45:45.523208 5588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:45:45 dist-test-slave-1nl6 krb5kdc[5237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322745, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.6.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:45.544054 5588 init.cc:377] Logged in from keytab as oryx/127.28.6.2@KRBTEST.COM (short username oryx)
I20260810 00:45:45.545672 5588 webserver.cc:460] Webserver started at http://127.28.6.2:42339/ using document root <none> and password file <none>
I20260810 00:45:45.546402 5588 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:45.546552 5588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:45.546867 5588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:45.549845 5588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-1/data/instance:
uuid: "672dbbeddd0a4ecdb51a8f256074d9b2"
format_stamp: "Formatted at 2026-08-10 00:45:45 on dist-test-slave-1nl6"
server_key: "09080e58dbe7c146e9eb359471822f9a"
server_key_iv: "39eecf399f0abfd0ccc5fa5ea89de8c2"
server_key_version: "encryptionkey@0"
I20260810 00:45:45.550647 5588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance:
uuid: "672dbbeddd0a4ecdb51a8f256074d9b2"
format_stamp: "Formatted at 2026-08-10 00:45:45 on dist-test-slave-1nl6"
server_key: "09080e58dbe7c146e9eb359471822f9a"
server_key_iv: "39eecf399f0abfd0ccc5fa5ea89de8c2"
server_key_version: "encryptionkey@0"
I20260810 00:45:45.555804 5588 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260810 00:45:45.559178 5612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:45.560413 5588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 00:45:45.560590 5588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-1/wal
uuid: "672dbbeddd0a4ecdb51a8f256074d9b2"
format_stamp: "Formatted at 2026-08-10 00:45:45 on dist-test-slave-1nl6"
server_key: "09080e58dbe7c146e9eb359471822f9a"
server_key_iv: "39eecf399f0abfd0ccc5fa5ea89de8c2"
server_key_version: "encryptionkey@0"
I20260810 00:45:45.560791 5588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:45.587435 5588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:45.591813 5588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:45.592111 5588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:45.593057 5588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:45:45.594405 5588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:45:45.594516 5588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:45.594667 5588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:45:45.594741 5588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:45.629056 5588 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.2:39127
I20260810 00:45:45.629110 5784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.2:39127 every 8 connection(s)
I20260810 00:45:45.630347 5588 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
I20260810 00:45:45.636610 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 5588
I20260810 00:45:45.636824 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance
I20260810 00:45:45.637318 28696 external_mini_cluster.cc:1428] Setting key 23222472f1cdeb6cc3c11fbe5ba805b0
Aug 10 00:45:45 dist-test-slave-1nl6 krb5kdc[5237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322745, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.6.2@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM
I20260810 00:45:45.652392 5794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:45.632948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:41785 (local address 127.28.6.62:39675)
0810 00:45:45.633450 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:45.633455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:45.634509 (+ 1054us) server_negotiation.cc:407] Connection header received
0810 00:45:45.636347 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:45.636356 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:45.636450 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:45.636577 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:45.638232 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:45.639141 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:45.641639 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:45.642049 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:45.642149 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:45.646553 (+ 4404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:45.646558 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:45.646562 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:45.646593 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:45.649316 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:45.650029 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:45.650035 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:45.650039 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:45.650183 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:45.650616 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:45.650622 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:45.650626 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:45.650887 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:45.651116 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:45.651777 (+ 661us) server_negotiation.cc:299] Negotiation successful
0810 00:45:45.651995 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":243,"threads_started":1}
I20260810 00:45:45.653290 5795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:45.633520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.28.6.2:41785)
0810 00:45:45.634332 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:45.634370 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:45.636090 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:45.636820 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:45.636829 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:45.637387 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:45.638051 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:45.638080 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:45.639354 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:45.639359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:45.641476 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:45.641503 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:45.642290 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:45.642300 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:45.642628 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:45.643369 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:45.643403 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:45.646307 (+ 2904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:45.649464 (+ 3157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:45.649473 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:45.649488 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:45.649907 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:45.650301 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:45.650307 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:45.650310 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:45.650499 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:45.651089 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:45.651097 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:45.651407 (+ 310us) client_negotiation.cc:769] Sending connection context
0810 00:45:45.651666 (+ 259us) client_negotiation.cc:240] Negotiation successful
0810 00:45:45.652185 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":303,"threads_started":1}
I20260810 00:45:45.654719 5788 heartbeater.cc:347] Connected to a master server at 127.28.6.62:39675
I20260810 00:45:45.655048 5788 heartbeater.cc:464] Registering TS with master...
I20260810 00:45:45.655753 5788 heartbeater.cc:511] Master 127.28.6.62:39675 requested a full tablet report, sending...
I20260810 00:45:45.657305 5292 ts_manager.cc:194] Registered new tserver with Master: 672dbbeddd0a4ecdb51a8f256074d9b2 (127.28.6.2:39127)
I20260810 00:45:45.658597 5292 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.6.2@KRBTEST.COM'} at 127.28.6.2:41785
WARNING: no policy specified for oryx/127.28.6.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.6.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.6.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.3.keytab.
Entry for principal oryx/127.28.6.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.3.keytab.
I20260810 00:45:45.687667 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.3:0
--local_ip_for_outbound_sockets=127.28.6.3
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:39675
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.3.keytab
--principal=oryx/127.28.6.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:37683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:45:46.015027 5800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:46.015311 5800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:46.015515 5800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:46.022672 5800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:46.022806 5800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:46.023025 5800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.3
I20260810 00:45:46.032997 5800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:37683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.3.keytab
--principal=oryx/127.28.6.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:39675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.3
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:46.034403 5800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:46.035811 5800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:46.048930 5807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:46.048902 5808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:46.049104 5811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:46.049674 5800 server_base.cc:1034] running on GCE node
I20260810 00:45:46.050206 5800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:46.051119 5800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:46.052357 5800 hybrid_clock.cc:648] HybridClock initialized: now 1786322746052313 us; error 31 us; skew 500 ppm
I20260810 00:45:46.052711 5800 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:45:46 dist-test-slave-1nl6 krb5kdc[5237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322746, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.6.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:46.056082 5800 init.cc:377] Logged in from keytab as oryx/127.28.6.3@KRBTEST.COM (short username oryx)
I20260810 00:45:46.057655 5800 webserver.cc:460] Webserver started at http://127.28.6.3:36433/ using document root <none> and password file <none>
I20260810 00:45:46.058293 5800 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:46.058424 5800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:46.058781 5800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:46.061720 5800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-2/data/instance:
uuid: "dc1d57df533e4f8a87d5f4595ec26f31"
format_stamp: "Formatted at 2026-08-10 00:45:46 on dist-test-slave-1nl6"
server_key: "182794bd83790740e6a7dbfd722d9f9a"
server_key_iv: "067cf13cf26cc2d6e529cfa0c82d7ec3"
server_key_version: "encryptionkey@0"
I20260810 00:45:46.062459 5800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance:
uuid: "dc1d57df533e4f8a87d5f4595ec26f31"
format_stamp: "Formatted at 2026-08-10 00:45:46 on dist-test-slave-1nl6"
server_key: "182794bd83790740e6a7dbfd722d9f9a"
server_key_iv: "067cf13cf26cc2d6e529cfa0c82d7ec3"
server_key_version: "encryptionkey@0"
I20260810 00:45:46.067979 5800 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260810 00:45:46.071256 5820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:46.072540 5800 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 00:45:46.072722 5800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-2/wal
uuid: "dc1d57df533e4f8a87d5f4595ec26f31"
format_stamp: "Formatted at 2026-08-10 00:45:46 on dist-test-slave-1nl6"
server_key: "182794bd83790740e6a7dbfd722d9f9a"
server_key_iv: "067cf13cf26cc2d6e529cfa0c82d7ec3"
server_key_version: "encryptionkey@0"
I20260810 00:45:46.072937 5800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:46.100466 5800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:46.104212 5800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:46.104521 5800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:46.105454 5800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:45:46.106760 5800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:45:46.106868 5800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:46.107065 5800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:45:46.107111 5800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:46.118752 5579 heartbeater.cc:503] Master 127.28.6.62:39675 was elected leader, sending a full tablet report...
I20260810 00:45:46.146363 5800 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.3:39253
I20260810 00:45:46.146456 5987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.3:39253 every 8 connection(s)
I20260810 00:45:46.148312 5800 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
I20260810 00:45:46.149901 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 5800
I20260810 00:45:46.150084 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance
I20260810 00:45:46.150627 28696 external_mini_cluster.cc:1428] Setting key 320dbe97a9532d6acc8df1d75807b5b0
Aug 10 00:45:46 dist-test-slave-1nl6 krb5kdc[5237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322746, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.6.3@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM
I20260810 00:45:46.176746 5995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:46.154090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:40931 (local address 127.28.6.62:39675)
0810 00:45:46.159705 (+ 5615us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:46.159715 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:46.159738 (+ 23us) server_negotiation.cc:407] Connection header received
0810 00:45:46.159811 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:46.159819 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:46.159965 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:46.160116 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:46.161598 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:46.162362 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:46.164299 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:46.165018 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:46.165172 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:46.169430 (+ 4258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:46.169439 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:46.169448 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:46.169480 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:46.173282 (+ 3802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:46.174360 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:46.174366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:46.174369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:46.174524 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:46.175051 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:46.175056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:46.175059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:46.175432 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:46.175549 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:46.176271 (+ 722us) server_negotiation.cc:299] Negotiation successful
0810 00:45:46.176463 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5466,"thread_start_us":5362,"threads_started":1}
I20260810 00:45:46.177908 5994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:46.153602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.28.6.3:40931)
0810 00:45:46.154493 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:46.154547 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:46.157332 (+ 2785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:46.160313 (+ 2981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:46.160334 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:46.160864 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:46.161417 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:46.161435 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:46.162525 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:46.162530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:46.164140 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:46.164182 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:46.165264 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:46.165272 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:46.165517 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:46.166252 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:46.166285 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:46.169211 (+ 2926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:46.173478 (+ 4267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:46.173491 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:46.173508 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:46.174204 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:46.174659 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:46.174664 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:46.174668 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:46.174872 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:46.175582 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:46.175591 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:46.175930 (+ 339us) client_negotiation.cc:769] Sending connection context
0810 00:45:46.176162 (+ 232us) client_negotiation.cc:240] Negotiation successful
0810 00:45:46.176662 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":2213,"threads_started":1}
I20260810 00:45:46.179397 5990 heartbeater.cc:347] Connected to a master server at 127.28.6.62:39675
I20260810 00:45:46.179716 5990 heartbeater.cc:464] Registering TS with master...
I20260810 00:45:46.180708 5990 heartbeater.cc:511] Master 127.28.6.62:39675 requested a full tablet report, sending...
I20260810 00:45:46.182176 5291 ts_manager.cc:194] Registered new tserver with Master: dc1d57df533e4f8a87d5f4595ec26f31 (127.28.6.3:39253)
I20260810 00:45:46.183096 5291 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.6.3@KRBTEST.COM'} at 127.28.6.3:40931
I20260810 00:45:46.192564 28696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.6.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.6.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.6.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.61.keytab.
Entry for principal oryx/127.28.6.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.61.keytab.
I20260810 00:45:46.244916 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.61:37239
--webserver_interface=127.28.6.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.61.keytab
--principal=oryx/127.28.6.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:37683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:39675,127.28.6.61:37239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:45:46.576889 5999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:46.577149 5999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:46.577241 5999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:46.584110 5999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:45:46.584205 5999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:46.584255 5999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:46.584314 5999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:45:46.584352 5999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:45:46.593853 5999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:37683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:39675,127.28.6.61:37239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.61.keytab
--principal=oryx/127.28.6.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.61:37239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.6.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
W20260810 00:45:46.594137 5999 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.28.6.62:39675,127.28.6.61:37239'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260810 00:45:46.595110 5999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:46.596364 5999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:46.607791 6005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:46.607829 6009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:46.607800 6007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:46.608724 5999 server_base.cc:1034] running on GCE node
I20260810 00:45:46.609484 5999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:46.610828 5999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:46.612083 5999 hybrid_clock.cc:648] HybridClock initialized: now 1786322746612023 us; error 33 us; skew 500 ppm
I20260810 00:45:46.612445 5999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:45:46 dist-test-slave-1nl6 krb5kdc[5237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322746, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.6.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:46.615680 5999 init.cc:377] Logged in from keytab as oryx/127.28.6.61@KRBTEST.COM (short username oryx)
I20260810 00:45:46.617221 5999 webserver.cc:460] Webserver started at http://127.28.6.61:35093/ using document root <none> and password file <none>
I20260810 00:45:46.617894 5999 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:46.618026 5999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:46.618351 5999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:46.621408 5999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-1/data/instance:
uuid: "952b9af13aac44daae6473c2d7a19566"
format_stamp: "Formatted at 2026-08-10 00:45:46 on dist-test-slave-1nl6"
server_key: "853df7a105a77795b948fc298acfe187"
server_key_iv: "0f793e87e97dec77f2d1960bd9d9e827"
server_key_version: "encryptionkey@0"
I20260810 00:45:46.622282 5999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-1/wal/instance:
uuid: "952b9af13aac44daae6473c2d7a19566"
format_stamp: "Formatted at 2026-08-10 00:45:46 on dist-test-slave-1nl6"
server_key: "853df7a105a77795b948fc298acfe187"
server_key_iv: "0f793e87e97dec77f2d1960bd9d9e827"
server_key_version: "encryptionkey@0"
I20260810 00:45:46.627614 5999 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260810 00:45:46.631734 6021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:46.633252 5999 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260810 00:45:46.633426 5999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-1/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-1/wal
uuid: "952b9af13aac44daae6473c2d7a19566"
format_stamp: "Formatted at 2026-08-10 00:45:46 on dist-test-slave-1nl6"
server_key: "853df7a105a77795b948fc298acfe187"
server_key_iv: "0f793e87e97dec77f2d1960bd9d9e827"
server_key_version: "encryptionkey@0"
I20260810 00:45:46.633657 5999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:46.649101 5999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:46.653113 5999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:46.653319 5999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:46.661093 5788 heartbeater.cc:503] Master 127.28.6.62:39675 was elected leader, sending a full tablet report...
I20260810 00:45:46.671991 5999 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.61:37239
I20260810 00:45:46.672024 6093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.61:37239 every 8 connection(s)
I20260810 00:45:46.673259 5999 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-1/data/info.pb
I20260810 00:45:46.675127 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 5999
I20260810 00:45:46.677748 6094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:46.688198 6094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 00:45:46 dist-test-slave-1nl6 krb5kdc[5237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.6.61@KRBTEST.COM
Aug 10 00:45:46 dist-test-slave-1nl6 krb5kdc[5237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322746, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.6.61@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM
I20260810 00:45:46.704552 6099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:46.677405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42734 (local address 127.28.6.61:37239)
0810 00:45:46.678390 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:46.678405 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:46.678458 (+ 53us) server_negotiation.cc:407] Connection header received
0810 00:45:46.680847 (+ 2389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:46.680876 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:46.681411 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:46.681864 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:46.683184 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:46.685459 (+ 2275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:46.687423 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:46.688593 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:46.688808 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:46.694869 (+ 6061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:46.694883 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:46.694931 (+ 48us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:46.694995 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:46.699735 (+ 4740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:46.700646 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:46.700657 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:46.700667 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:46.700922 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:46.701383 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:46.701387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:46.701389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:46.701904 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:46.702158 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:46.702651 (+ 493us) server_negotiation.cc:299] Negotiation successful
0810 00:45:46.703116 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":713,"thread_start_us":388,"threads_started":1}
I20260810 00:45:46.709867 6103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:46.691510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.0.0.1:53604)
0810 00:45:46.692273 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:46.692331 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:46.692504 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:46.693505 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:46.693517 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:46.694514 (+ 997us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:46.694989 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:46.695003 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:46.696759 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:46.696766 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:46.698850 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:46.698869 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:46.700071 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:46.700078 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:46.700275 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:46.700975 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:46.701029 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:46.703508 (+ 2479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:46.707550 (+ 4042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:46.707562 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:46.707570 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:46.707994 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:46.708379 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:46.708383 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:46.708385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:46.708506 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:46.709089 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:46.709094 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:46.709222 (+ 128us) client_negotiation.cc:769] Sending connection context
0810 00:45:46.709375 (+ 153us) client_negotiation.cc:240] Negotiation successful
0810 00:45:46.709595 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":293,"threads_started":1}
I20260810 00:45:46.709982 6104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:46.691760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53604 (local address 127.28.6.62:39675)
0810 00:45:46.692993 (+ 1233us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:46.693000 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:46.693019 (+ 19us) server_negotiation.cc:407] Connection header received
0810 00:45:46.693090 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:46.693095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:46.693227 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:46.693379 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:46.695209 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:46.696542 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:46.699105 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:46.699863 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:46.699984 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:46.703767 (+ 3783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:46.703774 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:46.703780 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:46.703827 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:46.707386 (+ 3559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:46.708141 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:46.708146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:46.708148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:46.708279 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:46.708612 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:46.708615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:46.708617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:46.708963 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:46.709086 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:46.709503 (+ 417us) server_negotiation.cc:299] Negotiation successful
0810 00:45:46.709684 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1068,"thread_start_us":938,"threads_started":1}
I20260810 00:45:46.711755 6094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 00:45:46 dist-test-slave-1nl6 krb5kdc[5237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322746, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.6.61@KRBTEST.COM for oryx/127.28.6.61@KRBTEST.COM
I20260810 00:45:46.723426 6103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:46.712568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:37239 (local address 127.0.0.1:42736)
0810 00:45:46.712757 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:46.712772 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:46.712852 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:46.713120 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:46.713122 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:46.713419 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:46.713542 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:46.713550 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:46.714521 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:46.714525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:46.715425 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:46.715436 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:46.716073 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:46.716076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:46.716280 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:46.716795 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:46.716811 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:46.718782 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:46.721467 (+ 2685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:46.721471 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:46.721473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:46.721812 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:46.722143 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:46.722146 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:46.722148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:46.722266 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:46.722690 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:46.722693 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:46.722806 (+ 113us) client_negotiation.cc:769] Sending connection context
0810 00:45:46.723003 (+ 197us) client_negotiation.cc:240] Negotiation successful
0810 00:45:46.723200 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 00:45:46.723627 6099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:46.712634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42736 (local address 127.28.6.61:37239)
0810 00:45:46.712828 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:46.712832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:46.712850 (+ 18us) server_negotiation.cc:407] Connection header received
0810 00:45:46.712950 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:46.712954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:46.713010 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:46.713089 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:46.713660 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:46.714412 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:46.715541 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:46.715951 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:46.716047 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:46.719040 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:46.719045 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:46.719048 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:46.719075 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:46.721347 (+ 2272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:46.721910 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:46.721913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:46.721916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:46.722048 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:46.722359 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:46.722362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:46.722364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:46.722594 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:46.722700 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:46.723079 (+ 379us) server_negotiation.cc:299] Negotiation successful
0810 00:45:46.723333 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":61}
I20260810 00:45:46.728402 6094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566: Bootstrap starting.
I20260810 00:45:46.731655 6094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:46.732726 6094 log.cc:824] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566: Log is configured to *not* fsync() on all Append() calls
I20260810 00:45:46.735469 6094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566: No bootstrap required, opened a new log
I20260810 00:45:46.742344 6094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } }
I20260810 00:45:46.742636 6094 raft_consensus.cc:374] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:46.742754 6094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 952b9af13aac44daae6473c2d7a19566, State: Initialized, Role: FOLLOWER
I20260810 00:45:46.743389 6094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } }
I20260810 00:45:46.744211 6106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } } }
I20260810 00:45:46.744544 6106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 00:45:46.745148 6094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 00:45:46.753474 6118 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 00:45:46.753610 6118 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 00:45:46.756615 5999 master_runner.cc:428] Detected that this master 952b9af13aac44daae6473c2d7a19566 is joining an existing cluster
I20260810 00:45:46.756721 5999 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.6.61:37239
I20260810 00:45:46.788247 6104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:46.760499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53610 (local address 127.28.6.62:39675)
0810 00:45:46.760683 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:46.760689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:46.771952 (+ 11263us) server_negotiation.cc:407] Connection header received
0810 00:45:46.772215 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:46.772240 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:46.772342 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:46.772418 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:46.773816 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:46.774967 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:46.777289 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:46.778000 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:46.778117 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:46.780849 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:46.780855 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:46.780860 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:46.780898 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:46.784927 (+ 4029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:46.785802 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:46.785808 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:46.785811 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:46.785974 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:46.786457 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:46.786462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:46.786465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:46.786829 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:46.786979 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:46.787618 (+ 639us) server_negotiation.cc:299] Negotiation successful
0810 00:45:46.787896 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 00:45:46.788605 6099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:46.760544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42744 (local address 127.28.6.61:37239)
0810 00:45:46.760766 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:46.760774 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:46.771458 (+ 10684us) server_negotiation.cc:407] Connection header received
0810 00:45:46.771629 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:46.771636 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:46.771719 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:46.771822 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:46.773353 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:46.774574 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:46.776911 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:46.777717 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:46.777824 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:46.780705 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:46.780713 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:46.780720 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:46.780759 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:46.785072 (+ 4313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:46.785981 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:46.785986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:46.785990 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:46.786137 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:46.786605 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:46.786612 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:46.786615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:46.787011 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:46.787199 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:46.787787 (+ 588us) server_negotiation.cc:299] Negotiation successful
0810 00:45:46.788031 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 00:45:46.789708 6127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:46.760911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:37239 (local address 127.0.0.1:42744)
0810 00:45:46.771326 (+ 10415us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:46.771350 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:46.771505 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:46.771989 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:46.771994 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:46.772722 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:46.773189 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:46.773203 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:46.774766 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:46.774774 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:46.776743 (+ 1969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:46.776761 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:46.778167 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:46.778177 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:46.778417 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:46.779124 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:46.779150 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:46.780526 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:46.785218 (+ 4692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:46.785225 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:46.785232 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:46.785860 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:46.786284 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:46.786292 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:46.786298 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:46.786496 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:46.787294 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:46.787301 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:46.787494 (+ 193us) client_negotiation.cc:769] Sending connection context
0810 00:45:46.788842 (+ 1348us) client_negotiation.cc:240] Negotiation successful
0810 00:45:46.789154 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10220,"mutex_wait_us":39,"thread_start_us":6999,"threads_started":1}
I20260810 00:45:46.790076 6126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:46.761017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.0.0.1:53610)
0810 00:45:46.771847 (+ 10830us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:46.771867 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:46.772049 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:46.772677 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:46.772683 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:46.773311 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:46.773644 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:46.773655 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:46.775172 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:46.775181 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:46.777090 (+ 1909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:46.777105 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:46.778500 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:46.778506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:46.778692 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:46.779361 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:46.779388 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:46.780618 (+ 1230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:46.785107 (+ 4489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:46.785112 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:46.785116 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:46.785637 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:46.786136 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:46.786140 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:46.786144 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:46.786303 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:46.787083 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:46.787088 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:46.787318 (+ 230us) client_negotiation.cc:769] Sending connection context
0810 00:45:46.788970 (+ 1652us) client_negotiation.cc:240] Negotiation successful
0810 00:45:46.789350 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10658,"mutex_wait_us":46,"thread_start_us":6609,"threads_started":1}
WARNING: no policy specified for oryx/127.28.6.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.6.60@KRBTEST.COM" created.
Aug 10 00:45:46 dist-test-slave-1nl6 krb5kdc[5237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322744, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.6.62@KRBTEST.COM for oryx/127.28.6.61@KRBTEST.COM
I20260810 00:45:46.816169 6099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:46.796731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42752 (local address 127.28.6.61:37239)
0810 00:45:46.796948 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:46.796956 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:46.797626 (+ 670us) server_negotiation.cc:407] Connection header received
0810 00:45:46.797999 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:46.798005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:46.798099 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:46.798204 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:46.799773 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:46.800913 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:46.803214 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:46.803966 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:46.804114 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:46.808748 (+ 4634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:46.808758 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:46.808765 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:46.808805 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:46.812617 (+ 3812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:46.813611 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:46.813619 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:46.813625 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:46.813819 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:46.814343 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:46.814349 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:46.814353 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:46.814703 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:46.814861 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:46.815529 (+ 668us) server_negotiation.cc:299] Negotiation successful
0810 00:45:46.815765 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 00:45:46.816247 6132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:46.796580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:37239 (local address 127.0.0.1:42752)
0810 00:45:46.797487 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:46.797530 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:46.797771 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:46.798297 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:46.798307 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:46.799245 (+ 938us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:46.799584 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:46.799601 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:46.801077 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:46.801085 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:46.803008 (+ 1923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:46.803028 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:46.804343 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:46.804350 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:46.804553 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:46.805326 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:46.805351 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:46.808520 (+ 3169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:46.812779 (+ 4259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:46.812795 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:46.812805 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:46.813450 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:46.813993 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:46.813998 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:46.814002 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:46.814187 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:46.814888 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:46.814921 (+ 33us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:46.815154 (+ 233us) client_negotiation.cc:769] Sending connection context
0810 00:45:46.815396 (+ 242us) client_negotiation.cc:240] Negotiation successful
0810 00:45:46.815738 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":488,"threads_started":1}
I20260810 00:45:46.818660 5291 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.6.61:37239: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "952b9af13aac44daae6473c2d7a19566"
member_type: NON_VOTER
last_known_addr {
host: "127.28.6.61"
port: 37239
}
attrs {
promote: true
}
}
dest_uuid: "e6e4f5d8a3384fbcbee45036aad474fa"
cas_config_opid_index: -1
I20260810 00:45:46.820001 5291 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: NON_VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: true } }
I20260810 00:45:46.823593 6138 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 952b9af13aac44daae6473c2d7a19566 (127.28.6.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: NON_VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: true } } }
I20260810 00:45:46.824779 6057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.6.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.60.keytab.
Entry for principal oryx/127.28.6.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.60.keytab.
I20260810 00:45:46.826830 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.60:42769
--webserver_interface=127.28.6.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.60.keytab
--principal=oryx/127.28.6.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:37683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:39675,127.28.6.61:37239,127.28.6.60:42769
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 00:45:46.827941 6138 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.6.61:37239
I20260810 00:45:46.829092 5999 master.cc:537] Master@127.28.6.61:37239 shutting down...
I20260810 00:45:46.829202 6138 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: NON_VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: true } } }
I20260810 00:45:46.829396 6138 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [sys.catalog]: This master's current role is: LEADER
I20260810 00:45:46.830292 6057 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [term 1 FOLLOWER]: Refusing update from remote peer e6e4f5d8a3384fbcbee45036aad474fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260810 00:45:46.831416 6137 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: NON_VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.001s
W20260810 00:45:46.835778 5275 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260810 00:45:46.847774 5275 consensus_peers.cc:487] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa -> Peer 952b9af13aac44daae6473c2d7a19566 (127.28.6.61:37239): Couldn't send request to peer 952b9af13aac44daae6473c2d7a19566. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 00:45:46.851441 5999 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:45:46.851981 5999 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:45:46.852317 5999 tablet_replica.cc:330] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566: stopping tablet replica
I20260810 00:45:46.869854 5999 master.cc:552] Master@127.28.6.61:37239 shutdown complete.
I20260810 00:45:46.869988 5999 master_runner.cc:305] Clearing existing system tablet
I20260810 00:45:46.870442 5999 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:46.870545 5999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:46.873616 5999 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 00:45:46.875348 6147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:46.875777 5999 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:46.876027 5999 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 00:45:46.876121 5999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-1/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-1/wal
uuid: "952b9af13aac44daae6473c2d7a19566"
format_stamp: "Formatted at 2026-08-10 00:45:46 on dist-test-slave-1nl6"
server_key: "853df7a105a77795b948fc298acfe187"
server_key_iv: "0f793e87e97dec77f2d1960bd9d9e827"
server_key_version: "encryptionkey@0"
I20260810 00:45:46.880774 5999 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:45:46.888207 5999 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 00:45:46.888386 5999 log.cc:1196] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260810 00:45:46.888934 5999 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566: Deleting consensus metadata
I20260810 00:45:46.889209 5999 master_runner.cc:315] Copying system tablet from 127.28.6.62:39675
I20260810 00:45:46.891868 5999 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566: tablet copy: Beginning tablet copy session from remote peer at address 127.28.6.62:39675
I20260810 00:45:46.909880 6155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:46.892730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.0.0.1:53616)
0810 00:45:46.895371 (+ 2641us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:46.895392 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:46.895557 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:46.895987 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:46.895994 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:46.896558 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:46.896876 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:46.896894 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:46.898383 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:46.898388 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:46.899939 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:46.899955 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:46.900995 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:46.901001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:46.901196 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:46.901940 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:46.901968 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:46.903048 (+ 1080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:46.906997 (+ 3949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:46.907002 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:46.907006 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:46.907491 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:46.907944 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:46.907949 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:46.907952 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:46.908146 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:46.908776 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:46.908781 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:46.908964 (+ 183us) client_negotiation.cc:769] Sending connection context
0810 00:45:46.909114 (+ 150us) client_negotiation.cc:240] Negotiation successful
0810 00:45:46.909433 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2495,"thread_start_us":2361,"threads_started":1}
I20260810 00:45:46.909962 6104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:46.892952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53616 (local address 127.28.6.62:39675)
0810 00:45:46.893183 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:46.893192 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:46.895513 (+ 2321us) server_negotiation.cc:407] Connection header received
0810 00:45:46.895701 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:46.895707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:46.895816 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:46.895929 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:46.897064 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:46.898216 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:46.900124 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:46.900830 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:46.900962 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:46.903272 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:46.903280 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:46.903285 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:46.903324 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:46.906830 (+ 3506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:46.907636 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:46.907642 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:46.907646 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:46.907814 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:46.908280 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:46.908285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:46.908289 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:46.908635 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:46.908796 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:46.909268 (+ 472us) server_negotiation.cc:299] Negotiation successful
0810 00:45:46.909498 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 00:45:46.911247 5321 tablet_copy_service.cc:140] P e6e4f5d8a3384fbcbee45036aad474fa: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 952b9af13aac44daae6473c2d7a19566 ({username='oryx', principal='oryx/127.28.6.61@KRBTEST.COM'} at 127.0.0.1:53616)
I20260810 00:45:46.911543 5321 tablet_copy_service.cc:161] P e6e4f5d8a3384fbcbee45036aad474fa: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 952b9af13aac44daae6473c2d7a19566 at {username='oryx', principal='oryx/127.28.6.61@KRBTEST.COM'} at 127.0.0.1:53616: session id = 952b9af13aac44daae6473c2d7a19566-00000000000000000000000000000000
I20260810 00:45:46.916121 5321 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 00:45:46.918334 5999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 00:45:46.925499 5999 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566: tablet copy: Starting download of 0 data blocks...
I20260810 00:45:46.925870 5999 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566: tablet copy: Starting download of 1 WAL segments...
I20260810 00:45:46.928896 5999 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 00:45:46.933286 5321 tablet_copy_service.cc:342] P e6e4f5d8a3384fbcbee45036aad474fa: Request end of tablet copy session 952b9af13aac44daae6473c2d7a19566-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.6.61@KRBTEST.COM'} at 127.0.0.1:53616
I20260810 00:45:46.933513 5321 tablet_copy_service.cc:433] P e6e4f5d8a3384fbcbee45036aad474fa: ending tablet copy session 952b9af13aac44daae6473c2d7a19566-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 952b9af13aac44daae6473c2d7a19566
I20260810 00:45:46.945616 5999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:46.948274 6157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:46.948791 6159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:46.953774 5999 server_base.cc:1034] running on GCE node
W20260810 00:45:46.954126 6163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:46.954598 5999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:46.955406 5999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:46.956624 5999 hybrid_clock.cc:648] HybridClock initialized: now 1786322746956560 us; error 28 us; skew 500 ppm
I20260810 00:45:46.956912 5999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:45:46 dist-test-slave-1nl6 krb5kdc[5237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322746, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.6.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:46.959438 5999 init.cc:377] Logged in from keytab as oryx/127.28.6.61@KRBTEST.COM (short username oryx)
I20260810 00:45:46.960676 5999 webserver.cc:460] Webserver started at http://127.28.6.61:46619/ using document root <none> and password file <none>
I20260810 00:45:46.961089 5999 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:46.961179 5999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:46.964658 5999 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 00:45:46.966673 6174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:46.967227 5999 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 00:45:46.967527 5999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-1/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-1/wal
uuid: "952b9af13aac44daae6473c2d7a19566"
format_stamp: "Formatted at 2026-08-10 00:45:46 on dist-test-slave-1nl6"
server_key: "853df7a105a77795b948fc298acfe187"
server_key_iv: "0f793e87e97dec77f2d1960bd9d9e827"
server_key_version: "encryptionkey@0"
I20260810 00:45:46.967782 5999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:46.981956 5999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:46.986063 5999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:47.013036 5999 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.61:37239
I20260810 00:45:47.013067 6254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.61:37239 every 8 connection(s)
I20260810 00:45:47.014283 5999 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-1/data/info.pb
I20260810 00:45:47.019691 6255 sys_catalog.cc:263] Verifying existing consensus state
I20260810 00:45:47.023629 6255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566: Bootstrap starting.
I20260810 00:45:47.039961 6255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 00:45:47.040458 6255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566: Bootstrap complete.
I20260810 00:45:47.041568 6255 raft_consensus.cc:348] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: NON_VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: true } }
I20260810 00:45:47.041844 6255 raft_consensus.cc:729] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 952b9af13aac44daae6473c2d7a19566, State: Initialized, Role: LEARNER
I20260810 00:45:47.042138 6255 consensus_queue.cc:260] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: NON_VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: true } }
I20260810 00:45:47.043298 6255 sys_catalog.cc:564] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:45:47.043082 6258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: NON_VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: true } } }
I20260810 00:45:47.043622 6258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [sys.catalog]: This master's current role is: LEARNER
I20260810 00:45:47.059477 6270 catalog_manager.cc:1259] Loaded cluster ID: 55a845b735744815b615e8a0736d9f90
I20260810 00:45:47.059649 6270 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566: loading cluster ID for follower catalog manager: success
I20260810 00:45:47.064127 6270 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566: acquiring CA information for follower catalog manager: success
I20260810 00:45:47.067076 6270 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 00:45:47.185564 5990 heartbeater.cc:503] Master 127.28.6.62:39675 was elected leader, sending a full tablet report...
W20260810 00:45:47.271919 6140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:47.272459 6140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:47.272670 6140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:47.281476 6140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:45:47.281605 6140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:47.281713 6140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:47.281819 6140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:45:47.281904 6140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:45:47.291962 6140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:37683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:39675,127.28.6.61:37239,127.28.6.60:42769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.60.keytab
--principal=oryx/127.28.6.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.60:42769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.6.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:47.293370 6140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:47.294668 6140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:47.308877 6279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:47.309106 6280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:47.309238 6282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:47.310751 6140 server_base.cc:1034] running on GCE node
I20260810 00:45:47.311933 6140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:47.313740 6140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:47.315035 6140 hybrid_clock.cc:648] HybridClock initialized: now 1786322747314921 us; error 51 us; skew 500 ppm
I20260810 00:45:47.315472 6140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:45:47 dist-test-slave-1nl6 krb5kdc[5237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322747, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.6.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:47.319656 6140 init.cc:377] Logged in from keytab as oryx/127.28.6.60@KRBTEST.COM (short username oryx)
I20260810 00:45:47.321506 6140 webserver.cc:460] Webserver started at http://127.28.6.60:46731/ using document root <none> and password file <none>
I20260810 00:45:47.322207 6140 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:47.322363 6140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:47.322701 6140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:47.327946 6140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-2/data/instance:
uuid: "e8bd3d99c782483f87add2f7cb6f1a4c"
format_stamp: "Formatted at 2026-08-10 00:45:47 on dist-test-slave-1nl6"
server_key: "a865e7d190a1319c8f7122749cd19ca0"
server_key_iv: "e6289ea96ee9c1c34554941562f503f8"
server_key_version: "encryptionkey@0"
I20260810 00:45:47.329054 6140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-2/wal/instance:
uuid: "e8bd3d99c782483f87add2f7cb6f1a4c"
format_stamp: "Formatted at 2026-08-10 00:45:47 on dist-test-slave-1nl6"
server_key: "a865e7d190a1319c8f7122749cd19ca0"
server_key_iv: "e6289ea96ee9c1c34554941562f503f8"
server_key_version: "encryptionkey@0"
I20260810 00:45:47.335551 6293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:47.326021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42756 (local address 127.28.6.61:37239)
0810 00:45:47.327265 (+ 1244us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:47.327269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:47.327287 (+ 18us) server_negotiation.cc:407] Connection header received
0810 00:45:47.327352 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:47.327356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:47.327444 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:47.327536 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:47.328275 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:47.329723 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:47.332741 (+ 3018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:47.334099 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:47.334223 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:47.334560 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:47.335013 (+ 453us) server_negotiation.cc:299] Negotiation successful
0810 00:45:47.335129 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1067,"thread_start_us":854,"threads_started":1}
I20260810 00:45:47.335742 6292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:47.325606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:37239 (local address 127.0.0.1:42756)
0810 00:45:47.326498 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:47.326518 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:47.326696 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:47.327609 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:47.327615 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:47.327649 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:47.328083 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:47.328099 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:47.329908 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:47.329914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:47.332532 (+ 2618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:47.332554 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:47.334438 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:47.334449 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:47.334798 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:47.334864 (+ 66us) client_negotiation.cc:769] Sending connection context
0810 00:45:47.334989 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 00:45:47.335130 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"thread_start_us":496,"threads_started":1}
I20260810 00:45:47.337350 6212 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260810 00:45:47.338117 6140 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.004s
I20260810 00:45:47.338030 6258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6e4f5d8a3384fbcbee45036aad474fa. Latest consensus state: current_term: 1 leader_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: NON_VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: true } } }
I20260810 00:45:47.338387 6258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [sys.catalog]: This master's current role is: LEARNER
I20260810 00:45:47.342500 6295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:47.344029 6140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 00:45:47.344230 6140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-2/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-2/wal
uuid: "e8bd3d99c782483f87add2f7cb6f1a4c"
format_stamp: "Formatted at 2026-08-10 00:45:47 on dist-test-slave-1nl6"
server_key: "a865e7d190a1319c8f7122749cd19ca0"
server_key_iv: "e6289ea96ee9c1c34554941562f503f8"
server_key_version: "encryptionkey@0"
I20260810 00:45:47.344465 6140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:47.372303 6140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:47.377185 6140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:47.377550 6140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:47.412608 6140 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.60:42769
I20260810 00:45:47.412724 6367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.60:42769 every 8 connection(s)
I20260810 00:45:47.414230 6140 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-2/data/info.pb
I20260810 00:45:47.418668 6368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:47.419399 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 6140
I20260810 00:45:47.431097 6368 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 00:45:47 dist-test-slave-1nl6 krb5kdc[5237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.6.60@KRBTEST.COM
Aug 10 00:45:47 dist-test-slave-1nl6 krb5kdc[5237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322747, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.6.60@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM
I20260810 00:45:47.445654 6374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:47.421823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43916 (local address 127.28.6.60:42769)
0810 00:45:47.423080 (+ 1257us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:47.423092 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:47.423141 (+ 49us) server_negotiation.cc:407] Connection header received
0810 00:45:47.425072 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:47.425098 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:47.425475 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:47.425879 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:47.428192 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:47.429484 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:47.431183 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:47.431877 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:47.432072 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:47.437805 (+ 5733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:47.437813 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:47.437823 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:47.437856 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:47.441376 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:47.442142 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:47.442151 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:47.442160 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:47.442382 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:47.442779 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:47.442784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:47.442788 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:47.443418 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:47.443644 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:47.444347 (+ 703us) server_negotiation.cc:299] Negotiation successful
0810 00:45:47.444678 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":968,"thread_start_us":626,"threads_started":1}
I20260810 00:45:47.450518 6375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:47.433944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60548 (local address 127.28.6.62:39675)
0810 00:45:47.434504 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:47.434509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:47.435742 (+ 1233us) server_negotiation.cc:407] Connection header received
0810 00:45:47.435976 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:47.435982 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:47.436057 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:47.436316 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:47.437511 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:47.438237 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:47.440454 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:47.440827 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:47.440923 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:47.445025 (+ 4102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:47.445030 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:47.445033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:47.445065 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:47.448082 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:47.448841 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:47.448845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:47.448847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:47.448970 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:47.449289 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:47.449292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:47.449294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:47.449583 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:47.449728 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:47.450105 (+ 377us) server_negotiation.cc:299] Negotiation successful
0810 00:45:47.450274 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":319,"threads_started":1}
I20260810 00:45:47.450578 6377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:47.434269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.0.0.1:60548)
0810 00:45:47.435590 (+ 1321us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:47.435626 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:47.435840 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:47.436384 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:47.436393 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:47.437093 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:47.437358 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:47.437366 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:47.438394 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:47.438398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:47.440304 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:47.440319 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:47.441020 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:47.441028 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:47.441242 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:47.441856 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:47.441872 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:47.444819 (+ 2947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:47.448238 (+ 3419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:47.448249 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:47.448256 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:47.448725 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:47.449065 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:47.449068 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:47.449070 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:47.449191 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:47.449736 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:47.449741 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:47.449868 (+ 127us) client_negotiation.cc:769] Sending connection context
0810 00:45:47.450006 (+ 138us) client_negotiation.cc:240] Negotiation successful
0810 00:45:47.450316 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1041,"thread_start_us":932,"threads_started":1}
I20260810 00:45:47.451926 6368 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 00:45:47 dist-test-slave-1nl6 krb5kdc[5237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322747, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.6.60@KRBTEST.COM for oryx/127.28.6.61@KRBTEST.COM
I20260810 00:45:47.466113 6377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:47.452627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:37239 (local address 127.0.0.1:39948)
0810 00:45:47.452821 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:47.452840 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:47.452934 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:47.453259 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:47.453263 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:47.453589 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:47.453715 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:47.453723 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:47.454675 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:47.454681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:47.455631 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:47.455642 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:47.456505 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:47.456513 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:47.456808 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:47.457371 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:47.457386 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:47.459673 (+ 2287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:47.463549 (+ 3876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:47.463553 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:47.463556 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:47.463985 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:47.464655 (+ 670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:47.464658 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:47.464660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:47.464842 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:47.465462 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:47.465465 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:47.465585 (+ 120us) client_negotiation.cc:769] Sending connection context
0810 00:45:47.465690 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 00:45:47.465904 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 00:45:47.466495 6293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:47.452709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39948 (local address 127.28.6.61:37239)
0810 00:45:47.452912 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:47.452918 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:47.452937 (+ 19us) server_negotiation.cc:407] Connection header received
0810 00:45:47.453037 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:47.453042 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:47.453113 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:47.453215 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:47.453856 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:47.454547 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:47.455769 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:47.456285 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:47.456427 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:47.459885 (+ 3458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:47.459890 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:47.459893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:47.459922 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:47.463436 (+ 3514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:47.464133 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:47.464137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:47.464139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:47.464549 (+ 410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:47.464945 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:47.464949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:47.464951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:47.465338 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:47.465552 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:47.465776 (+ 224us) server_negotiation.cc:299] Negotiation successful
0810 00:45:47.465999 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 00:45:47.467489 6368 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.6.60" port: 42769 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 00:45:47 dist-test-slave-1nl6 krb5kdc[5237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322747, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.6.60@KRBTEST.COM for oryx/127.28.6.60@KRBTEST.COM
I20260810 00:45:47.483793 6377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:47.468244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:42769 (local address 127.0.0.1:43924)
0810 00:45:47.468514 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:47.468530 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:47.468653 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:47.468946 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:47.468950 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:47.469432 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:47.469659 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:47.469668 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:47.470888 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:47.470896 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:47.472013 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:47.472025 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:47.472854 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:47.472859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:47.473072 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:47.473644 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:47.473670 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:47.476620 (+ 2950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:47.481075 (+ 4455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:47.481079 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:47.481082 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:47.481471 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:47.481915 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:47.481918 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:47.481920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:47.482162 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:47.482949 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:47.482952 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:47.483115 (+ 163us) client_negotiation.cc:769] Sending connection context
0810 00:45:47.483228 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 00:45:47.483395 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260810 00:45:47.484169 6374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:47.468296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43924 (local address 127.28.6.60:42769)
0810 00:45:47.468463 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:47.468474 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:47.468604 (+ 130us) server_negotiation.cc:407] Connection header received
0810 00:45:47.468773 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:47.468777 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:47.468837 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:47.468928 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:47.469797 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:47.470749 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:47.472165 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:47.472680 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:47.472835 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:47.476811 (+ 3976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:47.476817 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:47.476821 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:47.476854 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:47.480897 (+ 4043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:47.481602 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:47.481608 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:47.481612 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:47.481808 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:47.482300 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:47.482314 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:47.482318 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:47.482778 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:47.482938 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:47.483422 (+ 484us) server_negotiation.cc:299] Negotiation successful
0810 00:45:47.483701 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 00:45:47.491369 6368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c: Bootstrap starting.
I20260810 00:45:47.495800 6368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:47.497187 6368 log.cc:824] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c: Log is configured to *not* fsync() on all Append() calls
I20260810 00:45:47.500731 6368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c: No bootstrap required, opened a new log
I20260810 00:45:47.510155 6368 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } } peers { permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 42769 } }
I20260810 00:45:47.510514 6368 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:47.510702 6368 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8bd3d99c782483f87add2f7cb6f1a4c, State: Initialized, Role: FOLLOWER
I20260810 00:45:47.511332 6368 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } } peers { permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 42769 } }
I20260810 00:45:47.512712 6379 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } } peers { permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 42769 } } }
I20260810 00:45:47.513223 6379 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [sys.catalog]: This master's current role is: FOLLOWER
I20260810 00:45:47.513852 6368 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [sys.catalog]: configured and running, proceeding with master startup.
W20260810 00:45:47.527601 6395 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 00:45:47.527835 6395 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 00:45:47.532841 6140 master_runner.cc:428] Detected that this master e8bd3d99c782483f87add2f7cb6f1a4c is joining an existing cluster
I20260810 00:45:47.533077 6140 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.6.60:42769
I20260810 00:45:47.552050 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 5384
I20260810 00:45:47.569572 6375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:47.539425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60558 (local address 127.28.6.62:39675)
0810 00:45:47.539605 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:47.539613 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:47.545826 (+ 6213us) server_negotiation.cc:407] Connection header received
0810 00:45:47.546338 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:47.546347 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:47.546455 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:47.546545 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:47.550187 (+ 3642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:47.551594 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:47.555005 (+ 3411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:47.556275 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:47.556411 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:47.560668 (+ 4257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:47.560680 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:47.560688 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:47.560752 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:47.565355 (+ 4603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:47.566067 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:47.566073 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:47.566076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:47.566324 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:47.566991 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:47.566998 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:47.567002 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:47.567409 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:47.567569 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:47.568692 (+ 1123us) server_negotiation.cc:299] Negotiation successful
0810 00:45:47.569065 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 00:45:47.571950 6406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:47.539766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.0.0.1:60558)
0810 00:45:47.541506 (+ 1740us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:47.541532 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:47.546109 (+ 4577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:47.546840 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:47.546848 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:47.549136 (+ 2288us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:47.550036 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:47.550050 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:47.551863 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:47.551870 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:47.554736 (+ 2866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:47.554759 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:47.556904 (+ 2145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:47.556914 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:47.557318 (+ 404us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:47.558357 (+ 1039us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:47.558390 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:47.560352 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:47.565516 (+ 5164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:47.565521 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:47.565525 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:47.565931 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:47.566478 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:47.566486 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:47.566491 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:47.566808 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:47.567730 (+ 922us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:47.567737 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:47.568107 (+ 370us) client_negotiation.cc:769] Sending connection context
0810 00:45:47.569792 (+ 1685us) client_negotiation.cc:240] Negotiation successful
0810 00:45:47.570854 (+ 1062us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1522,"mutex_wait_us":69,"thread_start_us":1240,"threads_started":1}
Aug 10 00:45:47 dist-test-slave-1nl6 krb5kdc[5237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.6.1@KRBTEST.COM
I20260810 00:45:47.584858 6293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:47.539560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39950 (local address 127.28.6.61:37239)
0810 00:45:47.539719 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:47.539724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:47.546177 (+ 6453us) server_negotiation.cc:407] Connection header received
0810 00:45:47.546626 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:47.546634 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:47.546732 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:47.546806 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:47.553182 (+ 6376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:47.554584 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:47.557575 (+ 2991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:47.558235 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:47.558340 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:47.565526 (+ 7186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:47.565532 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:47.565537 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:47.565593 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:47.570402 (+ 4809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:47.572814 (+ 2412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:47.572821 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:47.572824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:47.573041 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:47.573855 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:47.573864 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:47.573871 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:47.574272 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:47.574424 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:47.583816 (+ 9392us) server_negotiation.cc:299] Negotiation successful
0810 00:45:47.584231 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 00:45:47.588403 6374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:47.541774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43930 (local address 127.28.6.60:42769)
0810 00:45:47.541966 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:47.541973 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:47.548944 (+ 6971us) server_negotiation.cc:407] Connection header received
0810 00:45:47.549151 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:47.549156 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:47.549221 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:47.549295 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:47.560840 (+ 11545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:47.562313 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:47.566485 (+ 4172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:47.567480 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:47.567574 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:47.576222 (+ 8648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:47.576228 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:47.576234 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:47.576302 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:47.582800 (+ 6498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:47.583739 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:47.583743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:47.583746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:47.583940 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:47.584486 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:47.584491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:47.584494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:47.585002 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:47.585153 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:47.585712 (+ 559us) server_negotiation.cc:299] Negotiation successful
0810 00:45:47.586372 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":385,"server-negotiator.queue_time_us":58}
I20260810 00:45:47.590142 6407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:47.542077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:42769 (local address 127.0.0.1:43930)
0810 00:45:47.548819 (+ 6742us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:47.548839 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:47.549032 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:47.549469 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:47.549473 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:47.551508 (+ 2035us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:47.560485 (+ 8977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:47.560501 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:47.563312 (+ 2811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:47.563318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:47.565309 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:47.565325 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:47.568218 (+ 2893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:47.568227 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:47.568459 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:47.570422 (+ 1963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:47.570854 (+ 432us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:47.575953 (+ 5099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:47.582968 (+ 7015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:47.582973 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:47.582976 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:47.583594 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:47.584078 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:47.584082 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:47.584085 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:47.584370 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:47.585221 (+ 851us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:47.585226 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:47.585395 (+ 169us) client_negotiation.cc:769] Sending connection context
0810 00:45:47.585804 (+ 409us) client_negotiation.cc:240] Negotiation successful
0810 00:45:47.586176 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6554,"thread_start_us":4139,"threads_started":1}
I20260810 00:45:47.598768 6409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:47.540600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:37239 (local address 127.0.0.1:39950)
0810 00:45:47.546042 (+ 5442us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:47.546061 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:47.546462 (+ 401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:47.547075 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:47.547080 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:47.549249 (+ 2169us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:47.552972 (+ 3723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:47.552988 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:47.554836 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:47.554842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:47.557387 (+ 2545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:47.557404 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:47.558641 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:47.558649 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:47.558958 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:47.559793 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:47.559832 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:47.563026 (+ 3194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:47.570530 (+ 7504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:47.570535 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:47.570539 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:47.572642 (+ 2103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:47.573424 (+ 782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:47.573428 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:47.573431 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:47.573703 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:47.582661 (+ 8958us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:47.582670 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:47.583403 (+ 733us) client_negotiation.cc:769] Sending connection context
0810 00:45:47.585439 (+ 2036us) client_negotiation.cc:240] Negotiation successful
0810 00:45:47.585981 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5269,"mutex_wait_us":62,"spinlock_wait_cycles":785152,"thread_start_us":7072,"threads_started":1}
I20260810 00:45:47.600056 6410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:47.554201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44936 (local address 127.28.6.1:42679)
0810 00:45:47.561888 (+ 7687us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:47.561897 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:47.561927 (+ 30us) server_negotiation.cc:407] Connection header received
0810 00:45:47.561993 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:47.562001 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:47.562291 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:47.562471 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:47.568204 (+ 5733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:47.570093 (+ 1889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:47.572364 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:47.573752 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:47.574558 (+ 806us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:47.588221 (+ 13663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:47.588243 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:47.588262 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:47.588305 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:47.594921 (+ 6616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:47.595953 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:47.595962 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:47.595966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:47.596180 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:47.596692 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:47.596698 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:47.596702 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:47.597183 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:47.597591 (+ 408us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:47.599187 (+ 1596us) server_negotiation.cc:299] Negotiation successful
0810 00:45:47.599531 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7479,"thread_start_us":7228,"threads_started":1}
I20260810 00:45:47.602416 5572 generic_service.cc:196] Checking for leaks (request via RPC)
Aug 10 00:45:47 dist-test-slave-1nl6 krb5kdc[5237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322744, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.6.62@KRBTEST.COM for oryx/127.28.6.60@KRBTEST.COM
I20260810 00:45:47.612950 6292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:47.578743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:42769 (local address 127.0.0.1:43936)
0810 00:45:47.578953 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:47.578987 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:47.579108 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:47.589715 (+ 10607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:47.589721 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:47.590359 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:47.590719 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:47.590734 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:47.594996 (+ 4262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:47.595005 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:47.596976 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:47.596993 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:47.599628 (+ 2635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:47.599634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:47.599850 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:47.600752 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:47.600777 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:47.603896 (+ 3119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:47.609822 (+ 5926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:47.609828 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:47.609832 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:47.610435 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:47.610958 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:47.610964 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:47.610968 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:47.611146 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:47.611851 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:47.611857 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:47.612054 (+ 197us) client_negotiation.cc:769] Sending connection context
0810 00:45:47.612225 (+ 171us) client_negotiation.cc:240] Negotiation successful
0810 00:45:47.612519 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 00:45:47.614261 6374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:47.579071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43936 (local address 127.28.6.60:42769)
0810 00:45:47.588986 (+ 9915us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:47.588995 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:47.589018 (+ 23us) server_negotiation.cc:407] Connection header received
0810 00:45:47.589134 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:47.589141 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:47.589262 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:47.589388 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:47.591129 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:47.594762 (+ 3633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:47.597175 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:47.598587 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:47.600517 (+ 1930us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:47.604617 (+ 4100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:47.604624 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:47.604629 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:47.604664 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:47.609618 (+ 4954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:47.610602 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:47.610607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:47.610611 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:47.610769 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:47.611308 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:47.611313 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:47.611316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:47.611702 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:47.613324 (+ 1622us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:47.613469 (+ 145us) server_negotiation.cc:299] Negotiation successful
0810 00:45:47.613716 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9784,"thread_start_us":20053,"threads_started":1}
I20260810 00:45:47.616076 5291 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.6.60:42769: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c"
member_type: NON_VOTER
last_known_addr {
host: "127.28.6.60"
port: 42769
}
attrs {
promote: true
}
}
dest_uuid: "e6e4f5d8a3384fbcbee45036aad474fa"
cas_config_opid_index: 5
I20260810 00:45:47.617409 5291 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: NON_VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: true } } peers { permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" member_type: NON_VOTER last_known_addr { host: "127.28.6.60" port: 42769 } attrs { promote: true } }
I20260810 00:45:47.619812 6138 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e8bd3d99c782483f87add2f7cb6f1a4c (127.28.6.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: NON_VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: true } } peers { permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" member_type: NON_VOTER last_known_addr { host: "127.28.6.60" port: 42769 } attrs { promote: true } } }
I20260810 00:45:47.623780 6333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:47.627907 6212 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [term 1 LEARNER]: Refusing update from remote peer e6e4f5d8a3384fbcbee45036aad474fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 00:45:47.632958 6414 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: NON_VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 00:45:47.639097 6138 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.6.60:42769
I20260810 00:45:47.639652 6138 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: NON_VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: true } } peers { permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" member_type: NON_VOTER last_known_addr { host: "127.28.6.60" port: 42769 } attrs { promote: true } } }
I20260810 00:45:47.639925 6138 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [sys.catalog]: This master's current role is: LEADER
W20260810 00:45:47.643388 6307 tablet.cc:2369] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:45:47.644683 6140 master.cc:537] Master@127.28.6.60:42769 shutting down...
I20260810 00:45:47.646399 6333 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [term 1 FOLLOWER]: Refusing update from remote peer e6e4f5d8a3384fbcbee45036aad474fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 00:45:47.647497 6415 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" member_type: NON_VOTER last_known_addr { host: "127.28.6.60" port: 42769 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:45:47.650688 6212 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e8bd3d99c782483f87add2f7cb6f1a4c (127.28.6.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: NON_VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: true } } peers { permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" member_type: NON_VOTER last_known_addr { host: "127.28.6.60" port: 42769 } attrs { promote: true } } }
W20260810 00:45:47.655180 5271 consensus_peers.cc:487] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa -> Peer e8bd3d99c782483f87add2f7cb6f1a4c (127.28.6.60:42769): Couldn't send request to peer e8bd3d99c782483f87add2f7cb6f1a4c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 00:45:47.657447 6140 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:45:47.657845 6140 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:45:47.658046 6140 tablet_replica.cc:330] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c: stopping tablet replica
I20260810 00:45:47.665012 6258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: NON_VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: true } } peers { permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" member_type: NON_VOTER last_known_addr { host: "127.28.6.60" port: 42769 } attrs { promote: true } } }
I20260810 00:45:47.665535 6140 master.cc:552] Master@127.28.6.60:42769 shutdown complete.
I20260810 00:45:47.665596 6258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [sys.catalog]: This master's current role is: LEARNER
I20260810 00:45:47.665694 6140 master_runner.cc:305] Clearing existing system tablet
I20260810 00:45:47.666234 6140 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:47.666448 6140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:47.666568 6414 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa: attempting to promote NON_VOTER 952b9af13aac44daae6473c2d7a19566 to VOTER
I20260810 00:45:47.667462 6414 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: false } } peers { permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" member_type: NON_VOTER last_known_addr { host: "127.28.6.60" port: 42769 } attrs { promote: true } }
I20260810 00:45:47.673761 6292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:47.673257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:42769 (local address 127.0.0.1:43944)
0810 00:45:47.673482 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:47.673594 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260810 00:45:47.674990 6292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:47.674524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:42769 (local address 127.0.0.1:43950)
0810 00:45:47.674738 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:47.674826 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:42769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260810 00:45:47.675156 6212 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [term 1 LEARNER]: Refusing update from remote peer e6e4f5d8a3384fbcbee45036aad474fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260810 00:45:47.675252 5271 consensus_peers.cc:487] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa -> Peer e8bd3d99c782483f87add2f7cb6f1a4c (127.28.6.60:42769): Couldn't send request to peer e8bd3d99c782483f87add2f7cb6f1a4c. Status: Network error: Client connection negotiation failed: client connection to 127.28.6.60:42769: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 00:45:47.675954 6138 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 00:45:47.677503 6140 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.002s sys 0.005s
I20260810 00:45:47.679160 6212 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 952b9af13aac44daae6473c2d7a19566 (127.28.6.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: false } } peers { permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" member_type: NON_VOTER last_known_addr { host: "127.28.6.60" port: 42769 } attrs { promote: true } } }
I20260810 00:45:47.679280 6420 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 952b9af13aac44daae6473c2d7a19566 (127.28.6.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: false } } peers { permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" member_type: NON_VOTER last_known_addr { host: "127.28.6.60" port: 42769 } attrs { promote: true } } }
I20260810 00:45:47.683609 6258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: false } } peers { permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" member_type: NON_VOTER last_known_addr { host: "127.28.6.60" port: 42769 } attrs { promote: true } } }
I20260810 00:45:47.683859 6258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 00:45:47.691232 6422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:47.695257 6140 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:47.695700 6140 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.006s sys 0.007s
I20260810 00:45:47.695860 6140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-2/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-2/wal
uuid: "e8bd3d99c782483f87add2f7cb6f1a4c"
format_stamp: "Formatted at 2026-08-10 00:45:47 on dist-test-slave-1nl6"
server_key: "a865e7d190a1319c8f7122749cd19ca0"
server_key_iv: "e6289ea96ee9c1c34554941562f503f8"
server_key_version: "encryptionkey@0"
I20260810 00:45:47.701131 6140 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:45:47.702308 6414 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: false } } peers { permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" member_type: NON_VOTER last_known_addr { host: "127.28.6.60" port: 42769 } attrs { promote: true } } }
I20260810 00:45:47.702752 6414 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [sys.catalog]: This master's current role is: LEADER
I20260810 00:45:47.703205 6414 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: false } } peers { permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" member_type: NON_VOTER last_known_addr { host: "127.28.6.60" port: 42769 } attrs { promote: true } } }
I20260810 00:45:47.703590 6414 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [sys.catalog]: This master's current role is: LEADER
I20260810 00:45:47.710348 6140 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 00:45:47.710654 6140 log.cc:1196] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260810 00:45:47.711408 6140 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c: Deleting consensus metadata
I20260810 00:45:47.711704 6140 master_runner.cc:315] Copying system tablet from 127.28.6.62:39675
I20260810 00:45:47.717293 6140 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c: tablet copy: Beginning tablet copy session from remote peer at address 127.28.6.62:39675
I20260810 00:45:47.736968 6375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:47.718184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60566 (local address 127.28.6.62:39675)
0810 00:45:47.718480 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:47.718491 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:47.719191 (+ 700us) server_negotiation.cc:407] Connection header received
0810 00:45:47.719388 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:47.719394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:47.719545 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:47.719668 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:47.720878 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:47.722414 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:47.724587 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:47.725452 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:47.725592 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:47.728405 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:47.728412 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:47.728417 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:47.728459 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:47.733187 (+ 4728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:47.734136 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:47.734142 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:47.734146 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:47.734381 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:47.734933 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:47.734938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:47.734942 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:47.735364 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:47.735590 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:47.736152 (+ 562us) server_negotiation.cc:299] Negotiation successful
0810 00:45:47.736394 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260810 00:45:47.737010 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:47.718201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.0.0.1:60566)
0810 00:45:47.719048 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:47.719082 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:47.719236 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:47.719714 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:47.719720 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:47.720425 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:47.720703 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:47.720715 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:47.722788 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:47.722805 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:47.724422 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:47.724438 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:47.725626 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:47.725632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:47.725826 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:47.726754 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:47.726776 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:47.728129 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:47.733368 (+ 5239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:47.733374 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:47.733378 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:47.733982 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:47.734511 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:47.734516 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:47.734520 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:47.734726 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:47.735523 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:47.735528 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:47.735813 (+ 285us) client_negotiation.cc:769] Sending connection context
0810 00:45:47.735996 (+ 183us) client_negotiation.cc:240] Negotiation successful
0810 00:45:47.736438 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":536,"threads_started":1}
I20260810 00:45:47.738188 5321 tablet_copy_service.cc:140] P e6e4f5d8a3384fbcbee45036aad474fa: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e8bd3d99c782483f87add2f7cb6f1a4c ({username='oryx', principal='oryx/127.28.6.60@KRBTEST.COM'} at 127.0.0.1:60566)
I20260810 00:45:47.738435 5321 tablet_copy_service.cc:161] P e6e4f5d8a3384fbcbee45036aad474fa: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e8bd3d99c782483f87add2f7cb6f1a4c at {username='oryx', principal='oryx/127.28.6.60@KRBTEST.COM'} at 127.0.0.1:60566: session id = e8bd3d99c782483f87add2f7cb6f1a4c-00000000000000000000000000000000
I20260810 00:45:47.743196 5321 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 00:45:47.745510 6140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 00:45:47.753947 6140 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c: tablet copy: Starting download of 0 data blocks...
I20260810 00:45:47.754417 6140 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c: tablet copy: Starting download of 1 WAL segments...
I20260810 00:45:47.757906 6140 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 00:45:47.763468 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 5384
I20260810 00:45:47.769640 5321 tablet_copy_service.cc:342] P e6e4f5d8a3384fbcbee45036aad474fa: Request end of tablet copy session e8bd3d99c782483f87add2f7cb6f1a4c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.6.60@KRBTEST.COM'} at 127.0.0.1:60566
I20260810 00:45:47.769837 5321 tablet_copy_service.cc:433] P e6e4f5d8a3384fbcbee45036aad474fa: ending tablet copy session e8bd3d99c782483f87add2f7cb6f1a4c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e8bd3d99c782483f87add2f7cb6f1a4c
I20260810 00:45:47.786839 6140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:47.789492 6436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:47.795392 6438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:47.801460 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.1:42679
--local_ip_for_outbound_sockets=127.28.6.1
--tserver_master_addrs=127.28.6.62:39675
--webserver_port=40279
--webserver_interface=127.28.6.1
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.1.keytab
--principal=oryx/127.28.6.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:37683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.6.62:39675,127.28.6.61:37239
--tserver_master_addrs=127.28.6.62:39675,127.28.6.61:37239,127.28.6.60:42769 with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:45:47.801546 6441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:47.804903 6140 server_base.cc:1034] running on GCE node
I20260810 00:45:47.805253 6140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:47.805960 6140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:47.807173 6140 hybrid_clock.cc:648] HybridClock initialized: now 1786322747807111 us; error 42 us; skew 500 ppm
I20260810 00:45:47.807464 6140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:45:47 dist-test-slave-1nl6 krb5kdc[5237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322747, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.6.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:47.809917 6140 init.cc:377] Logged in from keytab as oryx/127.28.6.60@KRBTEST.COM (short username oryx)
I20260810 00:45:47.811429 6140 webserver.cc:460] Webserver started at http://127.28.6.60:40749/ using document root <none> and password file <none>
I20260810 00:45:47.811777 6140 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:47.811893 6140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:47.815351 6140 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 00:45:47.817619 6455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:47.818089 6140 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260810 00:45:47.818264 6140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-2/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-2/wal
uuid: "e8bd3d99c782483f87add2f7cb6f1a4c"
format_stamp: "Formatted at 2026-08-10 00:45:47 on dist-test-slave-1nl6"
server_key: "a865e7d190a1319c8f7122749cd19ca0"
server_key_iv: "e6289ea96ee9c1c34554941562f503f8"
server_key_version: "encryptionkey@0"
I20260810 00:45:47.818529 6140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:47.858997 6140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:47.864526 6140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:47.897460 6140 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.60:42769
I20260810 00:45:47.897487 6534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.60:42769 every 8 connection(s)
I20260810 00:45:47.898574 6140 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/master-2/data/info.pb
I20260810 00:45:47.903843 6535 sys_catalog.cc:263] Verifying existing consensus state
I20260810 00:45:47.907923 6535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c: Bootstrap starting.
I20260810 00:45:47.927395 6535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 00:45:47.927965 6535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c: Bootstrap complete.
I20260810 00:45:47.929131 6535 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: false } } peers { permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" member_type: NON_VOTER last_known_addr { host: "127.28.6.60" port: 42769 } attrs { promote: true } }
I20260810 00:45:47.929451 6535 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e8bd3d99c782483f87add2f7cb6f1a4c, State: Initialized, Role: LEARNER
I20260810 00:45:47.929752 6535 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: false } } peers { permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" member_type: NON_VOTER last_known_addr { host: "127.28.6.60" port: 42769 } attrs { promote: true } }
I20260810 00:45:47.930619 6539 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: false } } peers { permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" member_type: NON_VOTER last_known_addr { host: "127.28.6.60" port: 42769 } attrs { promote: true } } }
I20260810 00:45:47.930965 6535 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:45:47.931135 6539 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [sys.catalog]: This master's current role is: LEARNER
I20260810 00:45:47.943430 6551 catalog_manager.cc:1259] Loaded cluster ID: 55a845b735744815b615e8a0736d9f90
I20260810 00:45:47.943507 6551 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c: loading cluster ID for follower catalog manager: success
I20260810 00:45:47.946271 6551 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c: acquiring CA information for follower catalog manager: success
I20260810 00:45:47.948108 6551 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 00:45:48.194108 6552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:48.186544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:42769 (local address 127.0.0.1:43956)
0810 00:45:48.187045 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:48.187061 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:48.187176 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:48.187958 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:48.187963 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:48.187992 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:48.188272 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:48.188284 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:48.190232 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:48.190239 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:48.191604 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:48.191618 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:48.192940 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:48.192946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:48.193127 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:48.193208 (+ 81us) client_negotiation.cc:769] Sending connection context
0810 00:45:48.193354 (+ 146us) client_negotiation.cc:240] Negotiation successful
0810 00:45:48.193532 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":258,"threads_started":1}
I20260810 00:45:48.194466 6554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:48.186791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43956 (local address 127.28.6.60:42769)
0810 00:45:48.187634 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:48.187639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:48.187656 (+ 17us) server_negotiation.cc:407] Connection header received
0810 00:45:48.187726 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:48.187730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:48.187797 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:48.187932 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:48.188461 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:48.190082 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:48.191765 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:48.192753 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:48.192903 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:48.193249 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:48.193375 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 00:45:48.193538 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":542,"threads_started":1}
I20260810 00:45:48.195664 6496 raft_consensus.cc:1203] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260810 00:45:48.196381 6539 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6e4f5d8a3384fbcbee45036aad474fa. Latest consensus state: current_term: 1 leader_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: false } } peers { permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" member_type: NON_VOTER last_known_addr { host: "127.28.6.60" port: 42769 } attrs { promote: true } } }
I20260810 00:45:48.196676 6539 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [sys.catalog]: This master's current role is: LEARNER
W20260810 00:45:48.266932 6447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:48.267359 6447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:48.267539 6447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:48.279260 6447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:48.279481 6447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:48.279757 6447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.1
I20260810 00:45:48.297457 6447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:37683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.1.keytab
--principal=oryx/127.28.6.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.1:42679
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.6.1
--webserver_port=40279
--tserver_master_addrs=127.28.6.62:39675,127.28.6.61:37239,127.28.6.60:42769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.1
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:48.299089 6447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:48.300531 6447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:48.317088 6559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:48.317209 6562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:48.317663 6560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:48.318465 6447 server_base.cc:1034] running on GCE node
I20260810 00:45:48.319288 6447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:48.320369 6447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:48.321659 6447 hybrid_clock.cc:648] HybridClock initialized: now 1786322748321557 us; error 50 us; skew 500 ppm
I20260810 00:45:48.322049 6447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:45:48 dist-test-slave-1nl6 krb5kdc[5237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322748, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.6.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:48.326443 6447 init.cc:377] Logged in from keytab as oryx/127.28.6.1@KRBTEST.COM (short username oryx)
I20260810 00:45:48.328398 6447 webserver.cc:460] Webserver started at http://127.28.6.1:40279/ using document root <none> and password file <none>
I20260810 00:45:48.329191 6447 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:48.329439 6447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:48.336081 6447 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260810 00:45:48.339360 6569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:48.340744 6447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 00:45:48.340963 6447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-0/wal
uuid: "6668f2f5a228466b8d3b7b8de4661e3d"
format_stamp: "Formatted at 2026-08-10 00:45:45 on dist-test-slave-1nl6"
server_key: "bfede56bf2b8249df9cdf93886b2fc75"
server_key_iv: "e83bb66301fec968f3d6c6293b6a61df"
server_key_version: "encryptionkey@0"
I20260810 00:45:48.342192 6447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:48.373015 6447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:48.378309 6447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:48.378662 6447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:48.379941 6447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:45:48.381668 6447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:45:48.381803 6447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:48.381899 6447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:45:48.381978 6447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:48.454020 6447 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.1:42679
I20260810 00:45:48.454107 6735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.1:42679 every 8 connection(s)
I20260810 00:45:48.455827 6447 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
I20260810 00:45:48.465445 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 6447
I20260810 00:45:48.465910 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 5588
Aug 10 00:45:48 dist-test-slave-1nl6 krb5kdc[5237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322748, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.6.1@KRBTEST.COM for oryx/127.28.6.61@KRBTEST.COM
Aug 10 00:45:48 dist-test-slave-1nl6 krb5kdc[5237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.6.2@KRBTEST.COM
Aug 10 00:45:48 dist-test-slave-1nl6 krb5kdc[5237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322748, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.6.1@KRBTEST.COM for oryx/127.28.6.60@KRBTEST.COM
I20260810 00:45:48.495864 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:48.459695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:47757 (local address 127.28.6.61:37239)
0810 00:45:48.460686 (+ 991us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:48.460694 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:48.470056 (+ 9362us) server_negotiation.cc:407] Connection header received
0810 00:45:48.470248 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:48.470254 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:48.470373 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:48.470497 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:48.474126 (+ 3629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:48.475189 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:48.477286 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:48.478399 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:48.478522 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:48.484520 (+ 5998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:48.484531 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:48.484538 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:48.484598 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:48.490289 (+ 5691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:48.491634 (+ 1345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:48.491639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:48.491644 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:48.491830 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:48.492552 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:48.492560 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:48.492567 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:48.493421 (+ 854us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:48.493565 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:48.494835 (+ 1270us) server_negotiation.cc:299] Negotiation successful
0810 00:45:48.495391 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":848,"thread_start_us":656,"threads_started":1}
I20260810 00:45:48.498832 6746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:48.460784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:37239 (local address 127.28.6.1:47757)
0810 00:45:48.469958 (+ 9174us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:48.469971 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:48.470118 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:48.470763 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:48.470772 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:48.472528 (+ 1756us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:48.473845 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:48.473900 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:48.475366 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:48.475377 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:48.477090 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:48.477102 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:48.479155 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:48.479172 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:48.479594 (+ 422us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:48.480636 (+ 1042us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:48.480673 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:48.484172 (+ 3499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:48.490494 (+ 6322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:48.490504 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:48.490522 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:48.491419 (+ 897us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:48.492012 (+ 593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:48.492017 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:48.492022 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:48.492373 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:48.493665 (+ 1292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:48.493675 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:48.494258 (+ 583us) client_negotiation.cc:769] Sending connection context
0810 00:45:48.496456 (+ 2198us) client_negotiation.cc:240] Negotiation successful
0810 00:45:48.497075 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5212,"mutex_wait_us":3,"thread_start_us":3425,"threads_started":1}
I20260810 00:45:48.501773 6743 heartbeater.cc:347] Connected to a master server at 127.28.6.61:37239
I20260810 00:45:48.502158 6743 heartbeater.cc:464] Registering TS with master...
I20260810 00:45:48.503163 6753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:48.467894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39114 (local address 127.28.6.2:39127)
0810 00:45:48.473223 (+ 5329us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:48.473229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:48.473253 (+ 24us) server_negotiation.cc:407] Connection header received
0810 00:45:48.473339 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:48.473347 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:48.473595 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:48.476920 (+ 3325us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:48.477259 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:48.478982 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:48.481675 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:48.482639 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:48.482822 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:48.491227 (+ 8405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:48.491244 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:48.491264 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:48.491350 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:48.497435 (+ 6085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:48.498716 (+ 1281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:48.498727 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:48.498734 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:48.498986 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:48.499490 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:48.499498 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:48.499501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:48.500484 (+ 983us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:48.500913 (+ 429us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:48.502248 (+ 1335us) server_negotiation.cc:299] Negotiation successful
0810 00:45:48.502573 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5148,"thread_start_us":4900,"threads_started":1}
I20260810 00:45:48.503556 6743 heartbeater.cc:511] Master 127.28.6.61:37239 requested a full tablet report, sending...
I20260810 00:45:48.504618 5774 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:45:48.506124 6554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:48.459497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:38569 (local address 127.28.6.60:42769)
0810 00:45:48.459716 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:48.459722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:48.466761 (+ 7039us) server_negotiation.cc:407] Connection header received
0810 00:45:48.469887 (+ 3126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:48.469897 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:48.470013 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:48.470146 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:48.482995 (+ 12849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:48.484434 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:48.487177 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:48.488039 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:48.488292 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:48.495905 (+ 7613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:48.495913 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:48.495919 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:48.495965 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:48.500509 (+ 4544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:48.501403 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:48.501408 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:48.501412 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:48.501574 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:48.502388 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:48.502393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:48.502396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:48.503547 (+ 1151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:48.503745 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:48.505007 (+ 1262us) server_negotiation.cc:299] Negotiation successful
0810 00:45:48.505461 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260810 00:45:48.506345 6748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:48.460269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:42769 (local address 127.28.6.1:38569)
0810 00:45:48.466428 (+ 6159us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:48.466518 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:48.469572 (+ 3054us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:48.471694 (+ 2122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:48.471702 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:48.473490 (+ 1788us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:48.474034 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:48.474049 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:48.484666 (+ 10617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:48.484672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:48.486949 (+ 2277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:48.486966 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:48.488646 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:48.488663 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:48.488870 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:48.491230 (+ 2360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:48.491253 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:48.495651 (+ 4398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:48.500658 (+ 5007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:48.500663 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:48.500668 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:48.501267 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:48.501715 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:48.501721 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:48.501725 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:48.502137 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:48.503810 (+ 1673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:48.503815 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:48.503966 (+ 151us) client_negotiation.cc:769] Sending connection context
0810 00:45:48.504117 (+ 151us) client_negotiation.cc:240] Negotiation successful
0810 00:45:48.504543 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3959,"mutex_wait_us":57,"thread_start_us":5283,"threads_started":1}
Aug 10 00:45:48 dist-test-slave-1nl6 krb5kdc[5237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322748, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.6.1@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM
I20260810 00:45:48.506732 6198 ts_manager.cc:194] Registered new tserver with Master: 6668f2f5a228466b8d3b7b8de4661e3d (127.28.6.1:42679)
I20260810 00:45:48.508646 6739 heartbeater.cc:347] Connected to a master server at 127.28.6.60:42769
I20260810 00:45:48.508852 6739 heartbeater.cc:464] Registering TS with master...
I20260810 00:45:48.509680 6739 heartbeater.cc:511] Master 127.28.6.60:42769 requested a full tablet report, sending...
I20260810 00:45:48.512353 6480 ts_manager.cc:194] Registered new tserver with Master: 6668f2f5a228466b8d3b7b8de4661e3d (127.28.6.1:42679)
I20260810 00:45:48.517567 6750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:48.466515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:48955 (local address 127.28.6.62:39675)
0810 00:45:48.471690 (+ 5175us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:48.471698 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:48.471720 (+ 22us) server_negotiation.cc:407] Connection header received
0810 00:45:48.471798 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:48.471803 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:48.471902 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:48.474196 (+ 2294us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:48.483881 (+ 9685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:48.485164 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:48.493480 (+ 8316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:48.494271 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:48.494366 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:48.507946 (+ 13580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:48.507953 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:48.507959 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:48.508016 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:48.514008 (+ 5992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:48.514752 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:48.514760 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:48.514766 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:48.514980 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:48.515632 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:48.515640 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:48.515647 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:48.516203 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:48.516368 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:48.516922 (+ 554us) server_negotiation.cc:299] Negotiation successful
0810 00:45:48.517121 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5027,"thread_start_us":4850,"threads_started":1}
I20260810 00:45:48.517648 6749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:48.466269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.28.6.1:48955)
0810 00:45:48.467300 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:48.467320 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:48.469572 (+ 2252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:48.472134 (+ 2562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:48.472138 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:48.472981 (+ 843us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:48.483656 (+ 10675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:48.483678 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:48.485584 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:48.485590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:48.490825 (+ 5235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:48.490840 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:48.498232 (+ 7392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:48.498245 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:48.498521 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:48.502929 (+ 4408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:48.502952 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:48.507541 (+ 4589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:48.514139 (+ 6598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:48.514144 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:48.514148 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:48.514612 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:48.515130 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:48.515137 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:48.515143 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:48.515467 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:48.516354 (+ 887us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:48.516362 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:48.516532 (+ 170us) client_negotiation.cc:769] Sending connection context
0810 00:45:48.516735 (+ 203us) client_negotiation.cc:240] Negotiation successful
0810 00:45:48.517082 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"mutex_wait_us":63,"thread_start_us":588,"threads_started":1}
I20260810 00:45:48.518653 6742 heartbeater.cc:347] Connected to a master server at 127.28.6.62:39675
I20260810 00:45:48.518883 6742 heartbeater.cc:464] Registering TS with master...
I20260810 00:45:48.519665 6742 heartbeater.cc:511] Master 127.28.6.62:39675 requested a full tablet report, sending...
I20260810 00:45:48.520988 5291 ts_manager.cc:194] Re-registered known tserver with Master: 6668f2f5a228466b8d3b7b8de4661e3d (127.28.6.1:42679)
I20260810 00:45:48.521863 5291 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.6.1@KRBTEST.COM'} at 127.28.6.1:48955
I20260810 00:45:48.601127 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 5588
I20260810 00:45:48.629601 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.2:39127
--local_ip_for_outbound_sockets=127.28.6.2
--tserver_master_addrs=127.28.6.62:39675
--webserver_port=42339
--webserver_interface=127.28.6.2
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.2.keytab
--principal=oryx/127.28.6.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:37683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.6.62:39675,127.28.6.61:37239
--tserver_master_addrs=127.28.6.62:39675,127.28.6.61:37239,127.28.6.60:42769 with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 00:45:48.706072 6424 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: false } } peers { permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" member_type: NON_VOTER last_known_addr { host: "127.28.6.60" port: 42769 } attrs { promote: true } } }
I20260810 00:45:48.706475 6424 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [sys.catalog]: This master's current role is: LEADER
I20260810 00:45:48.707945 6424 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa: attempting to promote NON_VOTER e8bd3d99c782483f87add2f7cb6f1a4c to VOTER
I20260810 00:45:48.708882 6424 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: false } } peers { permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 42769 } attrs { promote: false } }
I20260810 00:45:48.711907 6212 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [term 1 FOLLOWER]: Refusing update from remote peer e6e4f5d8a3384fbcbee45036aad474fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260810 00:45:48.712103 6496 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [term 1 LEARNER]: Refusing update from remote peer e6e4f5d8a3384fbcbee45036aad474fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260810 00:45:48.712636 6424 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260810 00:45:48.713024 6758 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 42769 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260810 00:45:48.716125 6758 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e8bd3d99c782483f87add2f7cb6f1a4c (127.28.6.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: false } } peers { permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 42769 } attrs { promote: false } } }
I20260810 00:45:48.719619 6212 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e8bd3d99c782483f87add2f7cb6f1a4c (127.28.6.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: false } } peers { permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 42769 } attrs { promote: false } } }
I20260810 00:45:48.720369 6424 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: false } } peers { permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 42769 } attrs { promote: false } } }
I20260810 00:45:48.720754 6424 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [sys.catalog]: This master's current role is: LEADER
I20260810 00:45:48.721045 6424 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: false } } peers { permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 42769 } attrs { promote: false } } }
I20260810 00:45:48.720443 6496 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e8bd3d99c782483f87add2f7cb6f1a4c (127.28.6.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: false } } peers { permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 42769 } attrs { promote: false } } }
I20260810 00:45:48.721293 6424 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [sys.catalog]: This master's current role is: LEADER
I20260810 00:45:48.721544 6758 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: false } } peers { permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 42769 } attrs { promote: false } } }
I20260810 00:45:48.721781 6758 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa [sys.catalog]: This master's current role is: LEADER
I20260810 00:45:48.725607 6763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: false } } peers { permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 42769 } attrs { promote: false } } }
I20260810 00:45:48.725965 6763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 00:45:48.726498 6764 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: false } } peers { permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 42769 } attrs { promote: false } } }
I20260810 00:45:48.726930 6764 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [sys.catalog]: This master's current role is: FOLLOWER
W20260810 00:45:48.982587 6755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:48.982859 6755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:48.983029 6755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:48.990115 6755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:48.990227 6755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:48.990346 6755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.2
I20260810 00:45:49.000092 6755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:37683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.2.keytab
--principal=oryx/127.28.6.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.2:39127
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.6.2
--webserver_port=42339
--tserver_master_addrs=127.28.6.62:39675,127.28.6.61:37239,127.28.6.60:42769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.2
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:49.001423 6755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:49.002743 6755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:49.017974 6775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:49.018553 6774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:49.018859 6777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:49.024066 6755 server_base.cc:1034] running on GCE node
I20260810 00:45:49.024748 6755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:49.025645 6755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:49.026854 6755 hybrid_clock.cc:648] HybridClock initialized: now 1786322749026786 us; error 54 us; skew 500 ppm
I20260810 00:45:49.027246 6755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:45:49 dist-test-slave-1nl6 krb5kdc[5237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322749, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.6.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:49.030503 6755 init.cc:377] Logged in from keytab as oryx/127.28.6.2@KRBTEST.COM (short username oryx)
I20260810 00:45:49.032258 6755 webserver.cc:460] Webserver started at http://127.28.6.2:42339/ using document root <none> and password file <none>
I20260810 00:45:49.032987 6755 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:49.033167 6755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:49.039088 6755 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 00:45:49.041839 6785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:49.043304 6755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 00:45:49.043495 6755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-1/wal
uuid: "672dbbeddd0a4ecdb51a8f256074d9b2"
format_stamp: "Formatted at 2026-08-10 00:45:45 on dist-test-slave-1nl6"
server_key: "09080e58dbe7c146e9eb359471822f9a"
server_key_iv: "39eecf399f0abfd0ccc5fa5ea89de8c2"
server_key_version: "encryptionkey@0"
I20260810 00:45:49.044551 6755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:49.064334 6755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:49.068960 6755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:49.069300 6755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:49.070235 6755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:45:49.071620 6755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:45:49.071745 6755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:49.071882 6755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:45:49.071970 6755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:49.106627 6755 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.2:39127
I20260810 00:45:49.106710 6954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.2:39127 every 8 connection(s)
I20260810 00:45:49.108172 6755 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
I20260810 00:45:49.113247 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 6755
I20260810 00:45:49.113595 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 5800
Aug 10 00:45:49 dist-test-slave-1nl6 krb5kdc[5237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.6.3@KRBTEST.COM
Aug 10 00:45:49 dist-test-slave-1nl6 krb5kdc[5237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322749, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.6.2@KRBTEST.COM for oryx/127.28.6.60@KRBTEST.COM
Aug 10 00:45:49 dist-test-slave-1nl6 krb5kdc[5237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322749, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.6.2@KRBTEST.COM for oryx/127.28.6.61@KRBTEST.COM
Aug 10 00:45:49 dist-test-slave-1nl6 krb5kdc[5237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322749, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.6.2@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM
I20260810 00:45:49.153597 6962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:49.111062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:54273 (local address 127.28.6.60:42769)
0810 00:45:49.119752 (+ 8690us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:49.119760 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:49.123455 (+ 3695us) server_negotiation.cc:407] Connection header received
0810 00:45:49.125738 (+ 2283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:49.125745 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:49.125836 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:49.125943 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:49.130100 (+ 4157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.131862 (+ 1762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.134355 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.135154 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.135314 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:49.141427 (+ 6113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:49.141435 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:49.141441 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:49.141484 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:49.147009 (+ 5525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:49.149848 (+ 2839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:49.149857 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:49.149864 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:49.150132 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:49.150820 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:49.150828 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:49.150831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:49.151343 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:49.151537 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:49.152771 (+ 1234us) server_negotiation.cc:299] Negotiation successful
0810 00:45:49.152998 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8529,"thread_start_us":7662,"threads_started":1}
I20260810 00:45:49.154448 6966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:49.115064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35198 (local address 127.28.6.3:39253)
0810 00:45:49.120294 (+ 5230us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:49.120302 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:49.120338 (+ 36us) server_negotiation.cc:407] Connection header received
0810 00:45:49.120413 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:49.120420 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:49.120633 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:49.120875 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:49.124020 (+ 3145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.125939 (+ 1919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.128303 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.129466 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.129626 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:49.138816 (+ 9190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:49.138835 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:49.138854 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:49.138921 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:49.147012 (+ 8091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:49.148204 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:49.148213 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:49.148216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:49.148452 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:49.151280 (+ 2828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:49.151289 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:49.151295 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:49.152067 (+ 772us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:49.152541 (+ 474us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:49.153921 (+ 1380us) server_negotiation.cc:299] Negotiation successful
0810 00:45:49.154116 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5029,"thread_start_us":5404,"threads_started":1}
I20260810 00:45:49.155537 6967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:49.115456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:42769 (local address 127.28.6.2:54273)
0810 00:45:49.123241 (+ 7785us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:49.123276 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:49.125360 (+ 2084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:49.126998 (+ 1638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:49.127003 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:49.128251 (+ 1248us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:49.129930 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:49.129942 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.132174 (+ 2232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.132178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:49.134180 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:49.134198 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.135533 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.135545 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:49.135991 (+ 446us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:49.136738 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:49.136780 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:49.141140 (+ 4360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:49.147269 (+ 6129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:49.147279 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:49.147708 (+ 429us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:49.149653 (+ 1945us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:49.150294 (+ 641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:49.150301 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:49.150308 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:49.150667 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:49.151639 (+ 972us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:49.151649 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:49.152184 (+ 535us) client_negotiation.cc:769] Sending connection context
0810 00:45:49.152654 (+ 470us) client_negotiation.cc:240] Negotiation successful
0810 00:45:49.153237 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7583,"mutex_wait_us":188,"spinlock_wait_cycles":35328,"thread_start_us":7346,"threads_started":1}
I20260810 00:45:49.156108 5981 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:45:49.157943 6955 heartbeater.cc:347] Connected to a master server at 127.28.6.60:42769
I20260810 00:45:49.158471 6955 heartbeater.cc:464] Registering TS with master...
I20260810 00:45:49.158759 6961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:49.110968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:59045 (local address 127.28.6.61:37239)
0810 00:45:49.122071 (+ 11103us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:49.122081 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:49.122109 (+ 28us) server_negotiation.cc:407] Connection header received
0810 00:45:49.126689 (+ 4580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:49.126694 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:49.126777 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:49.126872 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:49.130035 (+ 3163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.131661 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.137630 (+ 5969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.138594 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.138719 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:49.144843 (+ 6124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:49.144849 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:49.144853 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:49.144896 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:49.154428 (+ 9532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:49.155397 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:49.155414 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:49.155417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:49.155721 (+ 304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:49.156490 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:49.156497 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:49.156503 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:49.156981 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:49.157129 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:49.157942 (+ 813us) server_negotiation.cc:299] Negotiation successful
0810 00:45:49.158250 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10893,"thread_start_us":11161,"threads_started":1}
I20260810 00:45:49.158985 6963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:49.112300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:37239 (local address 127.28.6.2:59045)
0810 00:45:49.120250 (+ 7950us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:49.120288 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:49.125346 (+ 5058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:49.127113 (+ 1767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:49.127120 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:49.128241 (+ 1121us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:49.129785 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:49.129819 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.132783 (+ 2964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.132791 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:49.137476 (+ 4685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:49.137489 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.139174 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.139185 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:49.139481 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:49.140376 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:49.140401 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:49.144603 (+ 4202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:49.154606 (+ 10003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:49.154611 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:49.154614 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:49.155175 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:49.155898 (+ 723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:49.155903 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:49.155906 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:49.156287 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:49.157525 (+ 1238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:49.157528 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:49.157632 (+ 104us) client_negotiation.cc:769] Sending connection context
0810 00:45:49.157779 (+ 147us) client_negotiation.cc:240] Negotiation successful
0810 00:45:49.158212 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7603,"mutex_wait_us":157,"spinlock_wait_cycles":20736,"thread_start_us":7234,"threads_started":1}
I20260810 00:45:49.160109 6958 heartbeater.cc:347] Connected to a master server at 127.28.6.61:37239
I20260810 00:45:49.160347 6958 heartbeater.cc:464] Registering TS with master...
I20260810 00:45:49.161162 6958 heartbeater.cc:511] Master 127.28.6.61:37239 requested a full tablet report, sending...
I20260810 00:45:49.163301 6198 ts_manager.cc:194] Registered new tserver with Master: 672dbbeddd0a4ecdb51a8f256074d9b2 (127.28.6.2:39127)
I20260810 00:45:49.163601 6969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:49.123500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:44871 (local address 127.28.6.62:39675)
0810 00:45:49.124060 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:49.124070 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:49.124768 (+ 698us) server_negotiation.cc:407] Connection header received
0810 00:45:49.126181 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:49.126187 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:49.126262 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:49.126392 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:49.131031 (+ 4639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.132457 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.138470 (+ 6013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.139346 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.139470 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:49.149112 (+ 9642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:49.149119 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:49.149123 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:49.149180 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:49.153752 (+ 4572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:49.157205 (+ 3453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:49.157211 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:49.157216 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:49.157409 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:49.161536 (+ 4127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:49.161540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:49.161543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:49.162043 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:49.162184 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:49.162814 (+ 630us) server_negotiation.cc:299] Negotiation successful
0810 00:45:49.163084 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":284,"threads_started":1}
I20260810 00:45:49.164237 6970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:49.123324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.28.6.2:44871)
0810 00:45:49.124647 (+ 1323us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:49.124664 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:49.125689 (+ 1025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:49.126672 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:49.126684 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:49.128492 (+ 1808us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:49.130077 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:49.130094 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.134284 (+ 4190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.134291 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:49.138244 (+ 3953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:49.138270 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.140194 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.140216 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:49.141086 (+ 870us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:49.141884 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:49.141922 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:49.148938 (+ 7016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:49.155011 (+ 6073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:49.155016 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:49.155019 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:49.156562 (+ 1543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:49.161013 (+ 4451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:49.161021 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:49.161026 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:49.161390 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:49.162194 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:49.162201 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:49.162392 (+ 191us) client_negotiation.cc:769] Sending connection context
0810 00:45:49.162577 (+ 185us) client_negotiation.cc:240] Negotiation successful
0810 00:45:49.163037 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1190,"mutex_wait_us":2,"thread_start_us":976,"threads_started":1}
I20260810 00:45:49.160359 6955 heartbeater.cc:511] Master 127.28.6.60:42769 requested a full tablet report, sending...
I20260810 00:45:49.165171 6956 heartbeater.cc:347] Connected to a master server at 127.28.6.62:39675
I20260810 00:45:49.165407 6956 heartbeater.cc:464] Registering TS with master...
I20260810 00:45:49.165932 6480 ts_manager.cc:194] Registered new tserver with Master: 672dbbeddd0a4ecdb51a8f256074d9b2 (127.28.6.2:39127)
I20260810 00:45:49.166326 6956 heartbeater.cc:511] Master 127.28.6.62:39675 requested a full tablet report, sending...
I20260810 00:45:49.167579 5291 ts_manager.cc:194] Re-registered known tserver with Master: 672dbbeddd0a4ecdb51a8f256074d9b2 (127.28.6.2:39127)
I20260810 00:45:49.168768 5291 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.6.2@KRBTEST.COM'} at 127.28.6.2:44871
I20260810 00:45:49.245483 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 5800
I20260810 00:45:49.272056 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.3:39253
--local_ip_for_outbound_sockets=127.28.6.3
--tserver_master_addrs=127.28.6.62:39675
--webserver_port=36433
--webserver_interface=127.28.6.3
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.3.keytab
--principal=oryx/127.28.6.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:37683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.6.62:39675,127.28.6.61:37239
--tserver_master_addrs=127.28.6.62:39675,127.28.6.61:37239,127.28.6.60:42769 with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 00:45:49.524230 6742 heartbeater.cc:503] Master 127.28.6.62:39675 was elected leader, sending a full tablet report...
W20260810 00:45:49.641923 6972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:49.642247 6972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:49.642449 6972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:49.652279 6972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:49.652451 6972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:49.652714 6972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.3
I20260810 00:45:49.665539 6972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:37683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.3.keytab
--principal=oryx/127.28.6.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.3:39253
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.6.3
--webserver_port=36433
--tserver_master_addrs=127.28.6.62:39675,127.28.6.61:37239,127.28.6.60:42769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.3
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:49.666942 6972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:49.668699 6972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:49.682220 6978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:49.682253 6982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:49.682413 6988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:49.684306 6972 server_base.cc:1034] running on GCE node
I20260810 00:45:49.685249 6972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:49.686517 6972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:49.687753 6972 hybrid_clock.cc:648] HybridClock initialized: now 1786322749687659 us; error 44 us; skew 500 ppm
I20260810 00:45:49.688098 6972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:45:49 dist-test-slave-1nl6 krb5kdc[5237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322749, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.6.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:49.691215 6972 init.cc:377] Logged in from keytab as oryx/127.28.6.3@KRBTEST.COM (short username oryx)
I20260810 00:45:49.692667 6972 webserver.cc:460] Webserver started at http://127.28.6.3:36433/ using document root <none> and password file <none>
I20260810 00:45:49.693346 6972 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:49.693504 6972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:49.698872 6972 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 00:45:49.701517 7001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:49.702836 6972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 00:45:49.703053 6972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-2/wal
uuid: "dc1d57df533e4f8a87d5f4595ec26f31"
format_stamp: "Formatted at 2026-08-10 00:45:46 on dist-test-slave-1nl6"
server_key: "182794bd83790740e6a7dbfd722d9f9a"
server_key_iv: "067cf13cf26cc2d6e529cfa0c82d7ec3"
server_key_version: "encryptionkey@0"
I20260810 00:45:49.704046 6972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:49.723702 6972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:49.727854 6972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:49.728178 6972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:49.729117 6972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:45:49.730458 6972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:45:49.730579 6972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:49.730701 6972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:45:49.730785 6972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:49.767135 6972 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.3:39253
I20260810 00:45:49.767241 7168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.3:39253 every 8 connection(s)
I20260810 00:45:49.768834 6972 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
I20260810 00:45:49.775893 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 6972
Aug 10 00:45:49 dist-test-slave-1nl6 krb5kdc[5237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322749, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.6.3@KRBTEST.COM for oryx/127.28.6.61@KRBTEST.COM
Aug 10 00:45:49 dist-test-slave-1nl6 krb5kdc[5237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322749, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.6.3@KRBTEST.COM for oryx/127.28.6.60@KRBTEST.COM
Aug 10 00:45:49 dist-test-slave-1nl6 krb5kdc[5237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322749, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.6.3@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM
I20260810 00:45:49.825479 7180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:49.772591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:37239 (local address 127.28.6.3:40899)
0810 00:45:49.776795 (+ 4204us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:49.776855 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:49.780454 (+ 3599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:49.792127 (+ 11673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:49.792139 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:49.793189 (+ 1050us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:49.794298 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:49.794331 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.804018 (+ 9687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.804024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:49.805720 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:49.805739 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.807041 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.807052 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:49.807408 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:49.808550 (+ 1142us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:49.808593 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:49.813305 (+ 4712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:49.817866 (+ 4561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:49.817876 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:49.817896 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:49.818605 (+ 709us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:49.819492 (+ 887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:49.819499 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:49.819503 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:49.819785 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:49.820819 (+ 1034us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:49.820833 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:49.821648 (+ 815us) client_negotiation.cc:769] Sending connection context
0810 00:45:49.823381 (+ 1733us) client_negotiation.cc:240] Negotiation successful
0810 00:45:49.823975 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3814,"thread_start_us":38873,"threads_started":1}
I20260810 00:45:49.828423 7178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:49.772434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:40899 (local address 127.28.6.61:37239)
0810 00:45:49.791398 (+ 18964us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:49.791408 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:49.791439 (+ 31us) server_negotiation.cc:407] Connection header received
0810 00:45:49.791526 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:49.791536 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:49.791654 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:49.794563 (+ 2909us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:49.794793 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.796340 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.805945 (+ 9605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.806709 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.807898 (+ 1189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:49.813616 (+ 5718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:49.813624 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:49.813629 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:49.813674 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:49.817685 (+ 4011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:49.818801 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:49.818809 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:49.818813 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:49.819325 (+ 512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:49.819956 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:49.819962 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:49.819965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:49.820354 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:49.826261 (+ 5907us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:49.826414 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 00:45:49.826689 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18778,"thread_start_us":18657,"threads_started":1}
I20260810 00:45:49.830221 7173 heartbeater.cc:347] Connected to a master server at 127.28.6.61:37239
I20260810 00:45:49.830544 7186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:49.772591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:42769 (local address 127.28.6.3:38585)
0810 00:45:49.783764 (+ 11173us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:49.783789 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:49.783948 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:49.797534 (+ 13586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:49.797543 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:49.798373 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:49.798745 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:49.798762 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.800952 (+ 2190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.800962 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:49.803786 (+ 2824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:49.803806 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.812654 (+ 8848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.812663 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:49.812903 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:49.813926 (+ 1023us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:49.813957 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:49.819596 (+ 5639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:49.825197 (+ 5601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:49.825202 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:49.825205 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:49.826316 (+ 1111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:49.827745 (+ 1429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:49.827751 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:49.827756 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:49.827987 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:49.829107 (+ 1120us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:49.829113 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:49.829308 (+ 195us) client_negotiation.cc:769] Sending connection context
0810 00:45:49.829454 (+ 146us) client_negotiation.cc:240] Negotiation successful
0810 00:45:49.829852 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10981,"mutex_wait_us":38,"spinlock_wait_cycles":75648,"thread_start_us":5083,"threads_started":1}
I20260810 00:45:49.830780 7173 heartbeater.cc:464] Registering TS with master...
I20260810 00:45:49.832023 7175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:49.772042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:38585 (local address 127.28.6.60:42769)
0810 00:45:49.796924 (+ 24882us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:49.796933 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:49.796959 (+ 26us) server_negotiation.cc:407] Connection header received
0810 00:45:49.797039 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:49.797047 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:49.797142 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:49.797284 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:49.799017 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.800662 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.810756 (+ 10094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.811503 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.811629 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:49.819847 (+ 8218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:49.819854 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:49.819859 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:49.819898 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:49.825019 (+ 5121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:49.827332 (+ 2313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:49.827338 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:49.827342 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:49.827602 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:49.828125 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:49.828130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:49.828134 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:49.828895 (+ 761us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:49.829031 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:49.830989 (+ 1958us) server_negotiation.cc:299] Negotiation successful
0810 00:45:49.831398 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24703,"thread_start_us":26394,"threads_started":1}
I20260810 00:45:49.832639 7173 heartbeater.cc:511] Master 127.28.6.61:37239 requested a full tablet report, sending...
I20260810 00:45:49.833832 7169 heartbeater.cc:347] Connected to a master server at 127.28.6.60:42769
I20260810 00:45:49.834045 7169 heartbeater.cc:464] Registering TS with master...
I20260810 00:45:49.835060 7169 heartbeater.cc:511] Master 127.28.6.60:42769 requested a full tablet report, sending...
I20260810 00:45:49.835533 6198 ts_manager.cc:194] Registered new tserver with Master: dc1d57df533e4f8a87d5f4595ec26f31 (127.28.6.3:39253)
I20260810 00:45:49.836738 6480 ts_manager.cc:194] Registered new tserver with Master: dc1d57df533e4f8a87d5f4595ec26f31 (127.28.6.3:39253)
I20260810 00:45:49.840358 7179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:49.781352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.28.6.3:36191)
0810 00:45:49.811507 (+ 30155us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:49.811527 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:49.811769 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:49.812372 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:49.812377 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:49.813597 (+ 1220us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:49.813974 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:49.813989 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.815508 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.815512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:49.817135 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:49.817150 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.818469 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.818479 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:49.818692 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:49.821395 (+ 2703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:49.821419 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:49.827599 (+ 6180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:49.833896 (+ 6297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:49.833900 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:49.833903 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:49.837016 (+ 3113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:49.837632 (+ 616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:49.837636 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:49.837640 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:49.837940 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:49.838837 (+ 897us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:49.838843 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:49.839076 (+ 233us) client_negotiation.cc:769] Sending connection context
0810 00:45:49.839267 (+ 191us) client_negotiation.cc:240] Negotiation successful
0810 00:45:49.839629 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30035,"thread_start_us":2104,"threads_started":1}
I20260810 00:45:49.840603 7182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:49.779298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:36191 (local address 127.28.6.62:39675)
0810 00:45:49.781353 (+ 2055us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:49.781359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:49.811606 (+ 30247us) server_negotiation.cc:407] Connection header received
0810 00:45:49.811926 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:49.811931 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:49.812005 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:49.812084 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:49.814176 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.815328 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.817306 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.818051 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.818169 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:49.828745 (+ 10576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:49.828752 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:49.828757 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:49.828792 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:49.833672 (+ 4880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:49.837192 (+ 3520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:49.837197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:49.837201 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:49.837449 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:49.838128 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:49.838136 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:49.838140 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:49.838672 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:49.838841 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:49.839479 (+ 638us) server_negotiation.cc:299] Negotiation successful
0810 00:45:49.839768 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1931,"thread_start_us":1833,"threads_started":1}
I20260810 00:45:49.842330 7170 heartbeater.cc:347] Connected to a master server at 127.28.6.62:39675
I20260810 00:45:49.842574 7170 heartbeater.cc:464] Registering TS with master...
I20260810 00:45:49.843700 7170 heartbeater.cc:511] Master 127.28.6.62:39675 requested a full tablet report, sending...
I20260810 00:45:49.845188 5291 ts_manager.cc:194] Re-registered known tserver with Master: dc1d57df533e4f8a87d5f4595ec26f31 (127.28.6.3:39253)
I20260810 00:45:49.846809 5291 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.6.3@KRBTEST.COM'} at 127.28.6.3:36191
I20260810 00:45:49.849318 7193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:49.792370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43972 (local address 127.28.6.60:42769)
0810 00:45:49.804663 (+ 12293us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:49.804670 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:49.804687 (+ 17us) server_negotiation.cc:407] Connection header received
0810 00:45:49.804755 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:49.804760 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:49.804832 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:49.804941 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:49.814976 (+ 10035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.817656 (+ 2680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.820323 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.820973 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.821132 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:49.832539 (+ 11407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:49.832550 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:49.832555 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:49.832650 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:49.841621 (+ 8971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:49.843993 (+ 2372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:49.844000 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:49.844005 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:49.845420 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:49.846270 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:49.846275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:49.846278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:49.846722 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:49.848261 (+ 1539us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:49.848470 (+ 209us) server_negotiation.cc:299] Negotiation successful
0810 00:45:49.848765 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12161,"thread_start_us":13404,"threads_started":1}
I20260810 00:45:49.850384 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:49.796005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39966 (local address 127.28.6.61:37239)
0810 00:45:49.811297 (+ 15292us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:49.811305 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:49.811325 (+ 20us) server_negotiation.cc:407] Connection header received
0810 00:45:49.811377 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:49.811384 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:49.811462 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:49.811583 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:49.820788 (+ 9205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.822096 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.825628 (+ 3532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.826624 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.826798 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:49.838449 (+ 11651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:49.838458 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:49.838464 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:49.838501 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:49.844395 (+ 5894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:49.845959 (+ 1564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:49.845967 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:49.845973 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:49.846319 (+ 346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:49.846847 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:49.846853 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:49.846856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:49.847482 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:49.847822 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:49.849400 (+ 1578us) server_negotiation.cc:299] Negotiation successful
0810 00:45:49.849684 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15145,"thread_start_us":15023,"threads_started":1}
I20260810 00:45:49.851780 7189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:49.789906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60574 (local address 127.28.6.62:39675)
0810 00:45:49.802285 (+ 12379us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:49.802294 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:49.803376 (+ 1082us) server_negotiation.cc:407] Connection header received
0810 00:45:49.806134 (+ 2758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:49.806141 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:49.806225 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:49.806311 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:49.813292 (+ 6981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.814743 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.819011 (+ 4268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.819858 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.819981 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:49.838807 (+ 18826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:49.838814 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:49.838819 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:49.838858 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:49.846212 (+ 7354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:49.848047 (+ 1835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:49.848052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:49.848055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:49.848383 (+ 328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:49.849200 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:49.849215 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:49.849219 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:49.849666 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:49.849971 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:49.850953 (+ 982us) server_negotiation.cc:299] Negotiation successful
0810 00:45:49.851223 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12167,"thread_start_us":12084,"threads_started":1}
I20260810 00:45:49.856447 5291 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 00:45:49.866554 5291 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 00:45:49.902029 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:49.891168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:42679 (local address 127.0.0.1:44942)
0810 00:45:49.891691 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:49.891711 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:49.891833 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:49.893059 (+ 1226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:49.893064 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:49.893096 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:49.893517 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:49.893534 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.896025 (+ 2491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.896029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:49.897688 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:49.897706 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.900171 (+ 2465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.900181 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:49.900623 (+ 442us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:49.900665 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 00:45:49.900753 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 00:45:49.900885 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":60,"thread_start_us":4088,"threads_started":1}
I20260810 00:45:49.902199 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:49.891799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44942 (local address 127.28.6.1:42679)
0810 00:45:49.892392 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:49.892403 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:49.892434 (+ 31us) server_negotiation.cc:407] Connection header received
0810 00:45:49.892540 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:49.892549 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:49.892831 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:49.893756 (+ 925us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:49.894036 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.895842 (+ 1806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.897901 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.899758 (+ 1857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.901115 (+ 1357us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:49.901394 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:49.901477 (+ 83us) server_negotiation.cc:299] Negotiation successful
0810 00:45:49.901735 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":286,"threads_started":1}
I20260810 00:45:49.907168 6641 tablet_service.cc:1467] Processing CreateTablet for tablet 12acc6c849e34f47a34c498d86a7a5c9 (DEFAULT_TABLE table=test-table [id=b81c141d1c044a44ad9823ab0d61787a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:49.909286 6641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12acc6c849e34f47a34c498d86a7a5c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:49.913899 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:49.892202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:39253 (local address 127.0.0.1:35210)
0810 00:45:49.892854 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:49.892875 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:49.892994 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:49.899817 (+ 6823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:49.899833 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:49.899877 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:49.900993 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:49.901008 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.906489 (+ 5481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.906495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:49.909587 (+ 3092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:49.909604 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.912625 (+ 3021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.912632 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:49.912895 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:49.912952 (+ 57us) client_negotiation.cc:769] Sending connection context
0810 00:45:49.913059 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 00:45:49.913203 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":71,"thread_start_us":368,"threads_started":1}
I20260810 00:45:49.914104 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:49.892976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39118 (local address 127.28.6.2:39127)
0810 00:45:49.893478 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:49.893487 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:49.898454 (+ 4967us) server_negotiation.cc:407] Connection header received
0810 00:45:49.900617 (+ 2163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:49.900625 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:49.900864 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:49.901055 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:49.903390 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.904644 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.908492 (+ 3848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.910395 (+ 1903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.910512 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:49.910770 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:49.913439 (+ 2669us) server_negotiation.cc:299] Negotiation successful
0810 00:45:49.913641 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":245,"threads_started":1}
I20260810 00:45:49.915424 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:49.892622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:39127 (local address 127.0.0.1:39118)
0810 00:45:49.898332 (+ 5710us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:49.898350 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:49.900363 (+ 2013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:49.901400 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:49.901406 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:49.901440 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:49.903213 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:49.903229 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.904806 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.904812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:49.908322 (+ 3510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:49.908338 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.913046 (+ 4708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.913054 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:49.913210 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:49.913245 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 00:45:49.914451 (+ 1206us) client_negotiation.cc:240] Negotiation successful
0810 00:45:49.914593 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5583,"mutex_wait_us":547,"thread_start_us":4893,"threads_started":1}
I20260810 00:45:49.916283 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:49.893004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35210 (local address 127.28.6.3:39253)
0810 00:45:49.899147 (+ 6143us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:49.899156 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:49.899189 (+ 33us) server_negotiation.cc:407] Connection header received
0810 00:45:49.899288 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:49.899296 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:49.899533 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:49.899811 (+ 278us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:49.902017 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.906230 (+ 4213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.910855 (+ 4625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:49.912337 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:49.912501 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:49.912766 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:49.915713 (+ 2947us) server_negotiation.cc:299] Negotiation successful
0810 00:45:49.915885 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5925,"thread_start_us":5762,"threads_started":1}
I20260810 00:45:49.920917 7073 tablet_service.cc:1467] Processing CreateTablet for tablet 12acc6c849e34f47a34c498d86a7a5c9 (DEFAULT_TABLE table=test-table [id=b81c141d1c044a44ad9823ab0d61787a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:49.921277 6860 tablet_service.cc:1467] Processing CreateTablet for tablet 12acc6c849e34f47a34c498d86a7a5c9 (DEFAULT_TABLE table=test-table [id=b81c141d1c044a44ad9823ab0d61787a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:49.922650 7073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12acc6c849e34f47a34c498d86a7a5c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:49.923245 6860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12acc6c849e34f47a34c498d86a7a5c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:49.937057 7222 tablet_bootstrap.cc:492] T 12acc6c849e34f47a34c498d86a7a5c9 P 6668f2f5a228466b8d3b7b8de4661e3d: Bootstrap starting.
I20260810 00:45:49.942767 7223 tablet_bootstrap.cc:492] T 12acc6c849e34f47a34c498d86a7a5c9 P dc1d57df533e4f8a87d5f4595ec26f31: Bootstrap starting.
I20260810 00:45:49.944228 7224 tablet_bootstrap.cc:492] T 12acc6c849e34f47a34c498d86a7a5c9 P 672dbbeddd0a4ecdb51a8f256074d9b2: Bootstrap starting.
I20260810 00:45:49.948804 7223 tablet_bootstrap.cc:654] T 12acc6c849e34f47a34c498d86a7a5c9 P dc1d57df533e4f8a87d5f4595ec26f31: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:49.948997 7222 tablet_bootstrap.cc:654] T 12acc6c849e34f47a34c498d86a7a5c9 P 6668f2f5a228466b8d3b7b8de4661e3d: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:49.950237 7224 tablet_bootstrap.cc:654] T 12acc6c849e34f47a34c498d86a7a5c9 P 672dbbeddd0a4ecdb51a8f256074d9b2: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:49.950273 7223 log.cc:824] T 12acc6c849e34f47a34c498d86a7a5c9 P dc1d57df533e4f8a87d5f4595ec26f31: Log is configured to *not* fsync() on all Append() calls
I20260810 00:45:49.950924 7222 log.cc:824] T 12acc6c849e34f47a34c498d86a7a5c9 P 6668f2f5a228466b8d3b7b8de4661e3d: Log is configured to *not* fsync() on all Append() calls
I20260810 00:45:49.952383 7224 log.cc:824] T 12acc6c849e34f47a34c498d86a7a5c9 P 672dbbeddd0a4ecdb51a8f256074d9b2: Log is configured to *not* fsync() on all Append() calls
I20260810 00:45:49.954557 7223 tablet_bootstrap.cc:492] T 12acc6c849e34f47a34c498d86a7a5c9 P dc1d57df533e4f8a87d5f4595ec26f31: No bootstrap required, opened a new log
I20260810 00:45:49.955063 7223 ts_tablet_manager.cc:1397] T 12acc6c849e34f47a34c498d86a7a5c9 P dc1d57df533e4f8a87d5f4595ec26f31: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20260810 00:45:49.956264 7224 tablet_bootstrap.cc:492] T 12acc6c849e34f47a34c498d86a7a5c9 P 672dbbeddd0a4ecdb51a8f256074d9b2: No bootstrap required, opened a new log
I20260810 00:45:49.956300 7222 tablet_bootstrap.cc:492] T 12acc6c849e34f47a34c498d86a7a5c9 P 6668f2f5a228466b8d3b7b8de4661e3d: No bootstrap required, opened a new log
I20260810 00:45:49.956586 7222 ts_tablet_manager.cc:1397] T 12acc6c849e34f47a34c498d86a7a5c9 P 6668f2f5a228466b8d3b7b8de4661e3d: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.004s
I20260810 00:45:49.956596 7224 ts_tablet_manager.cc:1397] T 12acc6c849e34f47a34c498d86a7a5c9 P 672dbbeddd0a4ecdb51a8f256074d9b2: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20260810 00:45:49.965960 7222 raft_consensus.cc:348] T 12acc6c849e34f47a34c498d86a7a5c9 P 6668f2f5a228466b8d3b7b8de4661e3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6668f2f5a228466b8d3b7b8de4661e3d" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 42679 } } peers { permanent_uuid: "dc1d57df533e4f8a87d5f4595ec26f31" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39253 } } peers { permanent_uuid: "672dbbeddd0a4ecdb51a8f256074d9b2" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 39127 } }
I20260810 00:45:49.966449 7222 raft_consensus.cc:374] T 12acc6c849e34f47a34c498d86a7a5c9 P 6668f2f5a228466b8d3b7b8de4661e3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:49.966590 7222 raft_consensus.cc:729] T 12acc6c849e34f47a34c498d86a7a5c9 P 6668f2f5a228466b8d3b7b8de4661e3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6668f2f5a228466b8d3b7b8de4661e3d, State: Initialized, Role: FOLLOWER
I20260810 00:45:49.967344 7222 consensus_queue.cc:260] T 12acc6c849e34f47a34c498d86a7a5c9 P 6668f2f5a228466b8d3b7b8de4661e3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6668f2f5a228466b8d3b7b8de4661e3d" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 42679 } } peers { permanent_uuid: "dc1d57df533e4f8a87d5f4595ec26f31" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39253 } } peers { permanent_uuid: "672dbbeddd0a4ecdb51a8f256074d9b2" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 39127 } }
I20260810 00:45:49.969806 7222 ts_tablet_manager.cc:1428] T 12acc6c849e34f47a34c498d86a7a5c9 P 6668f2f5a228466b8d3b7b8de4661e3d: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260810 00:45:49.970830 7223 raft_consensus.cc:348] T 12acc6c849e34f47a34c498d86a7a5c9 P dc1d57df533e4f8a87d5f4595ec26f31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6668f2f5a228466b8d3b7b8de4661e3d" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 42679 } } peers { permanent_uuid: "dc1d57df533e4f8a87d5f4595ec26f31" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39253 } } peers { permanent_uuid: "672dbbeddd0a4ecdb51a8f256074d9b2" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 39127 } }
I20260810 00:45:49.971429 7223 raft_consensus.cc:374] T 12acc6c849e34f47a34c498d86a7a5c9 P dc1d57df533e4f8a87d5f4595ec26f31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:49.971679 7223 raft_consensus.cc:729] T 12acc6c849e34f47a34c498d86a7a5c9 P dc1d57df533e4f8a87d5f4595ec26f31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc1d57df533e4f8a87d5f4595ec26f31, State: Initialized, Role: FOLLOWER
I20260810 00:45:49.972556 7223 consensus_queue.cc:260] T 12acc6c849e34f47a34c498d86a7a5c9 P dc1d57df533e4f8a87d5f4595ec26f31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6668f2f5a228466b8d3b7b8de4661e3d" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 42679 } } peers { permanent_uuid: "dc1d57df533e4f8a87d5f4595ec26f31" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39253 } } peers { permanent_uuid: "672dbbeddd0a4ecdb51a8f256074d9b2" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 39127 } }
I20260810 00:45:49.976265 7224 raft_consensus.cc:348] T 12acc6c849e34f47a34c498d86a7a5c9 P 672dbbeddd0a4ecdb51a8f256074d9b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6668f2f5a228466b8d3b7b8de4661e3d" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 42679 } } peers { permanent_uuid: "dc1d57df533e4f8a87d5f4595ec26f31" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39253 } } peers { permanent_uuid: "672dbbeddd0a4ecdb51a8f256074d9b2" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 39127 } }
I20260810 00:45:49.976879 7224 raft_consensus.cc:374] T 12acc6c849e34f47a34c498d86a7a5c9 P 672dbbeddd0a4ecdb51a8f256074d9b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:49.977059 7224 raft_consensus.cc:729] T 12acc6c849e34f47a34c498d86a7a5c9 P 672dbbeddd0a4ecdb51a8f256074d9b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 672dbbeddd0a4ecdb51a8f256074d9b2, State: Initialized, Role: FOLLOWER
I20260810 00:45:49.977943 7224 consensus_queue.cc:260] T 12acc6c849e34f47a34c498d86a7a5c9 P 672dbbeddd0a4ecdb51a8f256074d9b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6668f2f5a228466b8d3b7b8de4661e3d" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 42679 } } peers { permanent_uuid: "dc1d57df533e4f8a87d5f4595ec26f31" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39253 } } peers { permanent_uuid: "672dbbeddd0a4ecdb51a8f256074d9b2" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 39127 } }
I20260810 00:45:49.978405 7170 heartbeater.cc:503] Master 127.28.6.62:39675 was elected leader, sending a full tablet report...
I20260810 00:45:49.979203 7223 ts_tablet_manager.cc:1428] T 12acc6c849e34f47a34c498d86a7a5c9 P dc1d57df533e4f8a87d5f4595ec26f31: Time spent starting tablet: real 0.024s user 0.014s sys 0.010s
I20260810 00:45:49.981956 6956 heartbeater.cc:503] Master 127.28.6.62:39675 was elected leader, sending a full tablet report...
I20260810 00:45:49.982769 7224 ts_tablet_manager.cc:1428] T 12acc6c849e34f47a34c498d86a7a5c9 P 672dbbeddd0a4ecdb51a8f256074d9b2: Time spent starting tablet: real 0.026s user 0.021s sys 0.001s
I20260810 00:45:50.035490 7229 raft_consensus.cc:482] T 12acc6c849e34f47a34c498d86a7a5c9 P dc1d57df533e4f8a87d5f4595ec26f31 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:45:50.035928 7229 raft_consensus.cc:504] T 12acc6c849e34f47a34c498d86a7a5c9 P dc1d57df533e4f8a87d5f4595ec26f31 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6668f2f5a228466b8d3b7b8de4661e3d" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 42679 } } peers { permanent_uuid: "dc1d57df533e4f8a87d5f4595ec26f31" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39253 } } peers { permanent_uuid: "672dbbeddd0a4ecdb51a8f256074d9b2" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 39127 } }
I20260810 00:45:50.038162 7229 leader_election.cc:288] T 12acc6c849e34f47a34c498d86a7a5c9 P dc1d57df533e4f8a87d5f4595ec26f31 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6668f2f5a228466b8d3b7b8de4661e3d (127.28.6.1:42679), 672dbbeddd0a4ecdb51a8f256074d9b2 (127.28.6.2:39127)
W20260810 00:45:50.040918 7174 tablet.cc:2369] T 12acc6c849e34f47a34c498d86a7a5c9 P dc1d57df533e4f8a87d5f4595ec26f31: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:45:50.047523 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:50.038772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:57071 (local address 127.28.6.2:39127)
0810 00:45:50.039047 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:50.039054 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:50.039073 (+ 19us) server_negotiation.cc:407] Connection header received
0810 00:45:50.039150 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:50.039159 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:50.039244 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:50.039340 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:50.040032 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:50.041395 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:50.044268 (+ 2873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:50.046245 (+ 1977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:50.046353 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:50.046449 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:50.046969 (+ 520us) server_negotiation.cc:299] Negotiation successful
0810 00:45:50.047122 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 00:45:50.047597 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:50.038445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:34141 (local address 127.28.6.1:42679)
0810 00:45:50.038686 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:50.038691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:50.039245 (+ 554us) server_negotiation.cc:407] Connection header received
0810 00:45:50.039411 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:50.039416 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:50.039494 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:50.039592 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:50.040269 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:50.041125 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:50.044256 (+ 3131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:50.046200 (+ 1944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:50.046353 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:50.046449 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:50.046987 (+ 538us) server_negotiation.cc:299] Negotiation successful
0810 00:45:50.047133 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 00:45:50.047624 7179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:50.038363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:39127 (local address 127.28.6.3:57071)
0810 00:45:50.038635 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:50.038657 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:50.038788 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:50.039390 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:50.039394 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:50.039425 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:50.039773 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:50.039794 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:50.041577 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:50.041582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:50.044058 (+ 2476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:50.044074 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:50.046420 (+ 2346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:50.046425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:50.046722 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:50.046807 (+ 85us) client_negotiation.cc:769] Sending connection context
0810 00:45:50.046918 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 00:45:50.047090 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":120}
I20260810 00:45:50.047624 7186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:50.038365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:42679 (local address 127.28.6.3:34141)
0810 00:45:50.039114 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:50.039131 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:50.039274 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:50.039672 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:50.039677 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:50.039699 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:50.040059 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:50.040073 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:50.041322 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:50.041327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:50.044071 (+ 2744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:50.044080 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:50.046406 (+ 2326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:50.046413 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:50.046765 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:50.046805 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 00:45:50.046916 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 00:45:50.047093 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":628}
I20260810 00:45:50.048851 6884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12acc6c849e34f47a34c498d86a7a5c9" candidate_uuid: "dc1d57df533e4f8a87d5f4595ec26f31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "672dbbeddd0a4ecdb51a8f256074d9b2" is_pre_election: true
I20260810 00:45:50.049599 6884 raft_consensus.cc:2454] T 12acc6c849e34f47a34c498d86a7a5c9 P 672dbbeddd0a4ecdb51a8f256074d9b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc1d57df533e4f8a87d5f4595ec26f31 in term 0.
I20260810 00:45:50.050422 7005 leader_election.cc:302] T 12acc6c849e34f47a34c498d86a7a5c9 P dc1d57df533e4f8a87d5f4595ec26f31 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 672dbbeddd0a4ecdb51a8f256074d9b2, dc1d57df533e4f8a87d5f4595ec26f31; no voters:
I20260810 00:45:50.051036 7229 raft_consensus.cc:2781] T 12acc6c849e34f47a34c498d86a7a5c9 P dc1d57df533e4f8a87d5f4595ec26f31 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 00:45:50.051263 7229 raft_consensus.cc:482] T 12acc6c849e34f47a34c498d86a7a5c9 P dc1d57df533e4f8a87d5f4595ec26f31 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 00:45:50.051419 7229 raft_consensus.cc:3037] T 12acc6c849e34f47a34c498d86a7a5c9 P dc1d57df533e4f8a87d5f4595ec26f31 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:50.051235 6671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12acc6c849e34f47a34c498d86a7a5c9" candidate_uuid: "dc1d57df533e4f8a87d5f4595ec26f31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6668f2f5a228466b8d3b7b8de4661e3d" is_pre_election: true
I20260810 00:45:50.051810 6671 raft_consensus.cc:2454] T 12acc6c849e34f47a34c498d86a7a5c9 P 6668f2f5a228466b8d3b7b8de4661e3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc1d57df533e4f8a87d5f4595ec26f31 in term 0.
I20260810 00:45:50.054596 7229 raft_consensus.cc:504] T 12acc6c849e34f47a34c498d86a7a5c9 P dc1d57df533e4f8a87d5f4595ec26f31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6668f2f5a228466b8d3b7b8de4661e3d" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 42679 } } peers { permanent_uuid: "dc1d57df533e4f8a87d5f4595ec26f31" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39253 } } peers { permanent_uuid: "672dbbeddd0a4ecdb51a8f256074d9b2" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 39127 } }
I20260810 00:45:50.055335 7229 leader_election.cc:288] T 12acc6c849e34f47a34c498d86a7a5c9 P dc1d57df533e4f8a87d5f4595ec26f31 [CANDIDATE]: Term 1 election: Requested vote from peers 6668f2f5a228466b8d3b7b8de4661e3d (127.28.6.1:42679), 672dbbeddd0a4ecdb51a8f256074d9b2 (127.28.6.2:39127)
I20260810 00:45:50.055835 6671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12acc6c849e34f47a34c498d86a7a5c9" candidate_uuid: "dc1d57df533e4f8a87d5f4595ec26f31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6668f2f5a228466b8d3b7b8de4661e3d"
I20260810 00:45:50.055938 6884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12acc6c849e34f47a34c498d86a7a5c9" candidate_uuid: "dc1d57df533e4f8a87d5f4595ec26f31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "672dbbeddd0a4ecdb51a8f256074d9b2"
I20260810 00:45:50.056128 6671 raft_consensus.cc:3037] T 12acc6c849e34f47a34c498d86a7a5c9 P 6668f2f5a228466b8d3b7b8de4661e3d [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:50.056238 6884 raft_consensus.cc:3037] T 12acc6c849e34f47a34c498d86a7a5c9 P 672dbbeddd0a4ecdb51a8f256074d9b2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:50.059211 6671 raft_consensus.cc:2454] T 12acc6c849e34f47a34c498d86a7a5c9 P 6668f2f5a228466b8d3b7b8de4661e3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc1d57df533e4f8a87d5f4595ec26f31 in term 1.
I20260810 00:45:50.059712 6884 raft_consensus.cc:2454] T 12acc6c849e34f47a34c498d86a7a5c9 P 672dbbeddd0a4ecdb51a8f256074d9b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc1d57df533e4f8a87d5f4595ec26f31 in term 1.
I20260810 00:45:50.059700 7003 leader_election.cc:302] T 12acc6c849e34f47a34c498d86a7a5c9 P dc1d57df533e4f8a87d5f4595ec26f31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6668f2f5a228466b8d3b7b8de4661e3d, dc1d57df533e4f8a87d5f4595ec26f31; no voters:
I20260810 00:45:50.060271 7229 raft_consensus.cc:2781] T 12acc6c849e34f47a34c498d86a7a5c9 P dc1d57df533e4f8a87d5f4595ec26f31 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:45:50.061088 7229 raft_consensus.cc:686] T 12acc6c849e34f47a34c498d86a7a5c9 P dc1d57df533e4f8a87d5f4595ec26f31 [term 1 LEADER]: Becoming Leader. State: Replica: dc1d57df533e4f8a87d5f4595ec26f31, State: Running, Role: LEADER
I20260810 00:45:50.061753 7229 consensus_queue.cc:237] T 12acc6c849e34f47a34c498d86a7a5c9 P dc1d57df533e4f8a87d5f4595ec26f31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6668f2f5a228466b8d3b7b8de4661e3d" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 42679 } } peers { permanent_uuid: "dc1d57df533e4f8a87d5f4595ec26f31" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39253 } } peers { permanent_uuid: "672dbbeddd0a4ecdb51a8f256074d9b2" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 39127 } }
I20260810 00:45:50.067894 5291 catalog_manager.cc:5495] T 12acc6c849e34f47a34c498d86a7a5c9 P dc1d57df533e4f8a87d5f4595ec26f31 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc1d57df533e4f8a87d5f4595ec26f31 (127.28.6.3). New cstate: current_term: 1 leader_uuid: "dc1d57df533e4f8a87d5f4595ec26f31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6668f2f5a228466b8d3b7b8de4661e3d" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 42679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc1d57df533e4f8a87d5f4595ec26f31" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "672dbbeddd0a4ecdb51a8f256074d9b2" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 39127 } health_report { overall_health: UNKNOWN } } }
I20260810 00:45:50.114154 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:50.106871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35226 (local address 127.28.6.3:39253)
0810 00:45:50.107159 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:50.107166 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:50.107522 (+ 356us) server_negotiation.cc:407] Connection header received
0810 00:45:50.107818 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:50.107823 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:50.107924 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:50.108078 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:45:50.108583 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:50.109542 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:50.111337 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:50.111888 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:50.112036 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:50.112681 (+ 645us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:45:50.113367 (+ 686us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:45:50.113534 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:50.113641 (+ 107us) server_negotiation.cc:299] Negotiation successful
0810 00:45:50.113728 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
W20260810 00:45:50.120354 6960 tablet.cc:2369] T 12acc6c849e34f47a34c498d86a7a5c9 P 672dbbeddd0a4ecdb51a8f256074d9b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:45:50.124979 6884 raft_consensus.cc:1261] T 12acc6c849e34f47a34c498d86a7a5c9 P 672dbbeddd0a4ecdb51a8f256074d9b2 [term 1 FOLLOWER]: Refusing update from remote peer dc1d57df533e4f8a87d5f4595ec26f31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:45:50.125118 6671 raft_consensus.cc:1261] T 12acc6c849e34f47a34c498d86a7a5c9 P 6668f2f5a228466b8d3b7b8de4661e3d [term 1 FOLLOWER]: Refusing update from remote peer dc1d57df533e4f8a87d5f4595ec26f31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:45:50.126201 7232 consensus_queue.cc:1035] T 12acc6c849e34f47a34c498d86a7a5c9 P dc1d57df533e4f8a87d5f4595ec26f31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "672dbbeddd0a4ecdb51a8f256074d9b2" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 39127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:45:50.126754 7229 consensus_queue.cc:1035] T 12acc6c849e34f47a34c498d86a7a5c9 P dc1d57df533e4f8a87d5f4595ec26f31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6668f2f5a228466b8d3b7b8de4661e3d" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 42679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:45:50.170647 5291 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:45:50.170939 5291 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:45:50.178437 5291 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6e4f5d8a3384fbcbee45036aad474fa: Sending DeleteTablet for 3 replicas of tablet 12acc6c849e34f47a34c498d86a7a5c9
I20260810 00:45:50.179939 6860 tablet_service.cc:1514] Processing DeleteTablet for tablet 12acc6c849e34f47a34c498d86a7a5c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:45:50 UTC) from {username='oryx', principal='oryx/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:39118
I20260810 00:45:50.179988 7073 tablet_service.cc:1514] Processing DeleteTablet for tablet 12acc6c849e34f47a34c498d86a7a5c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:45:50 UTC) from {username='oryx', principal='oryx/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:35210
I20260810 00:45:50.180349 6641 tablet_service.cc:1514] Processing DeleteTablet for tablet 12acc6c849e34f47a34c498d86a7a5c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:45:50 UTC) from {username='oryx', principal='oryx/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:44942
I20260810 00:45:50.192441 7255 tablet_replica.cc:330] T 12acc6c849e34f47a34c498d86a7a5c9 P 6668f2f5a228466b8d3b7b8de4661e3d: stopping tablet replica
I20260810 00:45:50.193146 7255 raft_consensus.cc:2229] T 12acc6c849e34f47a34c498d86a7a5c9 P 6668f2f5a228466b8d3b7b8de4661e3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:45:50.194437 7255 raft_consensus.cc:2258] T 12acc6c849e34f47a34c498d86a7a5c9 P 6668f2f5a228466b8d3b7b8de4661e3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:45:50.196924 7255 ts_tablet_manager.cc:1905] T 12acc6c849e34f47a34c498d86a7a5c9 P 6668f2f5a228466b8d3b7b8de4661e3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:45:50.205430 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 5253
I20260810 00:45:50.207002 5337 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:45:50.208659 7255 ts_tablet_manager.cc:1918] T 12acc6c849e34f47a34c498d86a7a5c9 P 6668f2f5a228466b8d3b7b8de4661e3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:45:50.208943 7255 log.cc:1196] T 12acc6c849e34f47a34c498d86a7a5c9 P 6668f2f5a228466b8d3b7b8de4661e3d: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-0/wal/wals/12acc6c849e34f47a34c498d86a7a5c9
I20260810 00:45:50.211025 7254 tablet_replica.cc:330] T 12acc6c849e34f47a34c498d86a7a5c9 P 672dbbeddd0a4ecdb51a8f256074d9b2: stopping tablet replica
I20260810 00:45:50.211735 7254 raft_consensus.cc:2229] T 12acc6c849e34f47a34c498d86a7a5c9 P 672dbbeddd0a4ecdb51a8f256074d9b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:45:50.212622 7254 raft_consensus.cc:2258] T 12acc6c849e34f47a34c498d86a7a5c9 P 672dbbeddd0a4ecdb51a8f256074d9b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:45:50.212670 7255 ts_tablet_manager.cc:1939] T 12acc6c849e34f47a34c498d86a7a5c9 P 6668f2f5a228466b8d3b7b8de4661e3d: Deleting consensus metadata
I20260810 00:45:50.220059 7253 tablet_replica.cc:330] T 12acc6c849e34f47a34c498d86a7a5c9 P dc1d57df533e4f8a87d5f4595ec26f31: stopping tablet replica
I20260810 00:45:50.220916 7253 raft_consensus.cc:2229] T 12acc6c849e34f47a34c498d86a7a5c9 P dc1d57df533e4f8a87d5f4595ec26f31 [term 1 LEADER]: Raft consensus shutting down.
I20260810 00:45:50.221555 7253 raft_consensus.cc:2258] T 12acc6c849e34f47a34c498d86a7a5c9 P dc1d57df533e4f8a87d5f4595ec26f31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:45:50.223672 7254 ts_tablet_manager.cc:1905] T 12acc6c849e34f47a34c498d86a7a5c9 P 672dbbeddd0a4ecdb51a8f256074d9b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:45:50.223897 7253 ts_tablet_manager.cc:1905] T 12acc6c849e34f47a34c498d86a7a5c9 P dc1d57df533e4f8a87d5f4595ec26f31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:45:50.235123 7253 ts_tablet_manager.cc:1918] T 12acc6c849e34f47a34c498d86a7a5c9 P dc1d57df533e4f8a87d5f4595ec26f31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:45:50.235548 7253 log.cc:1196] T 12acc6c849e34f47a34c498d86a7a5c9 P dc1d57df533e4f8a87d5f4595ec26f31: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-2/wal/wals/12acc6c849e34f47a34c498d86a7a5c9
I20260810 00:45:50.236608 7253 ts_tablet_manager.cc:1939] T 12acc6c849e34f47a34c498d86a7a5c9 P dc1d57df533e4f8a87d5f4595ec26f31: Deleting consensus metadata
I20260810 00:45:50.239264 7254 ts_tablet_manager.cc:1918] T 12acc6c849e34f47a34c498d86a7a5c9 P 672dbbeddd0a4ecdb51a8f256074d9b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:45:50.239442 7254 log.cc:1196] T 12acc6c849e34f47a34c498d86a7a5c9 P 672dbbeddd0a4ecdb51a8f256074d9b2: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-1/wal/wals/12acc6c849e34f47a34c498d86a7a5c9
I20260810 00:45:50.239996 7254 ts_tablet_manager.cc:1939] T 12acc6c849e34f47a34c498d86a7a5c9 P 672dbbeddd0a4ecdb51a8f256074d9b2: Deleting consensus metadata
I20260810 00:45:50.249473 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:50.184430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39970 (local address 127.28.6.61:37239)
0810 00:45:50.184693 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:50.184701 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:50.203472 (+ 18771us) server_negotiation.cc:407] Connection header received
0810 00:45:50.203849 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:50.203855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:50.204006 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:50.204143 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:50.209495 (+ 5352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:50.211032 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:50.224378 (+ 13346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:50.225437 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:50.225608 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:50.238869 (+ 13261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:50.238876 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:50.238882 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:50.238956 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:50.245433 (+ 6477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:50.246353 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:50.246360 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:50.246364 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:50.246585 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:50.247192 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:50.247200 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:50.247205 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:50.247754 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:50.248179 (+ 425us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:50.248655 (+ 476us) server_negotiation.cc:299] Negotiation successful
0810 00:45:50.248920 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 00:45:50.272907 7193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:50.204285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43980 (local address 127.28.6.60:42769)
0810 00:45:50.204513 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:50.204519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:50.249587 (+ 45068us) server_negotiation.cc:407] Connection header received
0810 00:45:50.249667 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:50.249672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:50.249816 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:50.249925 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:50.253049 (+ 3124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:50.254297 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:50.256839 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:50.257860 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:50.257982 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:50.263946 (+ 5964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:50.263953 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:50.263959 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:50.263998 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:50.269136 (+ 5138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:50.270139 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:50.270144 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:50.270148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:50.270335 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:50.270924 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:50.270930 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:50.270934 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:50.271413 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:50.271685 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:50.272242 (+ 557us) server_negotiation.cc:299] Negotiation successful
0810 00:45:50.272491 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 00:45:50.361403 5270 catalog_manager.cc:4841] TS dc1d57df533e4f8a87d5f4595ec26f31 (127.28.6.3:39253): tablet 12acc6c849e34f47a34c498d86a7a5c9 (table test-table [id=b81c141d1c044a44ad9823ab0d61787a]) successfully deleted
I20260810 00:45:50.361980 5275 catalog_manager.cc:4841] TS 672dbbeddd0a4ecdb51a8f256074d9b2 (127.28.6.2:39127): tablet 12acc6c849e34f47a34c498d86a7a5c9 (table test-table [id=b81c141d1c044a44ad9823ab0d61787a]) successfully deleted
I20260810 00:45:50.361455 5271 catalog_manager.cc:4841] TS 6668f2f5a228466b8d3b7b8de4661e3d (127.28.6.1:42679): tablet 12acc6c849e34f47a34c498d86a7a5c9 (table test-table [id=b81c141d1c044a44ad9823ab0d61787a]) successfully deleted
W20260810 00:45:50.362412 5271 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.28.6.1:42679, user_credentials={real_user=slave}} blocked reactor thread for 147323us
I20260810 00:45:50.362813 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 5253
I20260810 00:45:50.418690 7193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:50.401386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43992 (local address 127.28.6.60:42769)
0810 00:45:50.401579 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:50.401585 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:50.401879 (+ 294us) server_negotiation.cc:407] Connection header received
0810 00:45:50.402149 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:50.402172 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:50.402282 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:50.402374 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:50.404433 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:50.405353 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:50.407449 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:50.408043 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:50.408198 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:50.412393 (+ 4195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:50.412398 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:50.412402 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:50.412433 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:50.415507 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:50.416482 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:50.416488 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:50.416491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:50.416628 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:50.417128 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:50.417138 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:50.417140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:50.417427 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:50.417677 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:50.418036 (+ 359us) server_negotiation.cc:299] Negotiation successful
0810 00:45:50.418245 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 00:45:50.419445 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:50.400044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39984 (local address 127.28.6.61:37239)
0810 00:45:50.400348 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:50.400357 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:50.401270 (+ 913us) server_negotiation.cc:407] Connection header received
0810 00:45:50.402184 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:50.402191 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:50.402293 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:50.402365 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:50.405023 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:50.405883 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:50.407554 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:50.408198 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:50.408332 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:50.412026 (+ 3694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:50.412033 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:50.412038 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:50.412077 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:50.415705 (+ 3628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:50.416513 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:50.416519 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:50.416523 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:50.416672 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:50.417106 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:50.417111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:50.417115 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:50.417493 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:50.417728 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:50.418855 (+ 1127us) server_negotiation.cc:299] Negotiation successful
0810 00:45:50.419094 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260810 00:45:51.151657 7274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:51.150957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.28.6.1:39769)
0810 00:45:51.151377 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:51.151496 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":209,"threads_started":1}
W20260810 00:45:51.152097 6742 heartbeater.cc:644] Failed to heartbeat to 127.28.6.62:39675 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
I20260810 00:45:51.164716 7275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:51.163898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.28.6.3:45227)
0810 00:45:51.164371 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:51.164535 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":211,"threads_started":1}
W20260810 00:45:51.165131 7170 heartbeater.cc:644] Failed to heartbeat to 127.28.6.62:39675 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
I20260810 00:45:51.166263 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:51.165591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.28.6.2:45049)
0810 00:45:51.165967 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:51.166087 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":188,"threads_started":1}
W20260810 00:45:51.166885 6956 heartbeater.cc:644] Failed to heartbeat to 127.28.6.62:39675 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
I20260810 00:45:51.678982 7281 raft_consensus.cc:482] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e6e4f5d8a3384fbcbee45036aad474fa)
I20260810 00:45:51.679406 7282 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e6e4f5d8a3384fbcbee45036aad474fa)
I20260810 00:45:51.679350 7281 raft_consensus.cc:504] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: false } } peers { permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 42769 } attrs { promote: false } }
I20260810 00:45:51.679682 7282 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: false } } peers { permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 42769 } attrs { promote: false } }
I20260810 00:45:51.680763 7281 leader_election.cc:288] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e6e4f5d8a3384fbcbee45036aad474fa (127.28.6.62:39675), e8bd3d99c782483f87add2f7cb6f1a4c (127.28.6.60:42769)
I20260810 00:45:51.681201 7282 leader_election.cc:288] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e6e4f5d8a3384fbcbee45036aad474fa (127.28.6.62:39675), 952b9af13aac44daae6473c2d7a19566 (127.28.6.61:37239)
I20260810 00:45:51.682363 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:51.680939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.0.0.1:60968)
0810 00:45:51.681873 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:51.682039 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":798,"thread_start_us":643,"threads_started":1}
I20260810 00:45:51.684166 7285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:51.681229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.0.0.1:60972)
0810 00:45:51.683622 (+ 2393us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:51.683851 (+ 229us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2219,"threads_started":1}
W20260810 00:45:51.685918 6458 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
W20260810 00:45:51.686066 6177 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
I20260810 00:45:51.695732 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:51.682020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:42769 (local address 127.0.0.1:44008)
0810 00:45:51.682751 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:51.682782 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:51.682973 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:51.685637 (+ 2664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:51.685645 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:51.685684 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:51.686006 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:51.686021 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:51.688264 (+ 2243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:51.688274 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:51.691110 (+ 2836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:51.691132 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:51.694646 (+ 3514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:51.694655 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:51.694931 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:51.694988 (+ 57us) client_negotiation.cc:769] Sending connection context
0810 00:45:51.695086 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 00:45:51.695277 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":3191,"threads_started":1}
I20260810 00:45:51.696923 7290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:51.683582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44008 (local address 127.28.6.60:42769)
0810 00:45:51.685285 (+ 1703us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:51.685290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:51.685307 (+ 17us) server_negotiation.cc:407] Connection header received
0810 00:45:51.685372 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:51.685377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:51.685448 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:51.686169 (+ 721us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:51.686396 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:51.688048 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:51.691381 (+ 3333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:51.694411 (+ 3030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:51.695959 (+ 1548us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:51.696127 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:51.696194 (+ 67us) server_negotiation.cc:299] Negotiation successful
0810 00:45:51.696397 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":1551,"thread_start_us":1356,"threads_started":1}
I20260810 00:45:51.697518 7285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:51.681407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:37239 (local address 127.0.0.1:40000)
0810 00:45:51.684503 (+ 3096us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:51.684525 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:51.684680 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:51.686492 (+ 1812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:51.686499 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:51.686537 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:51.686937 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:51.686954 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:51.689279 (+ 2325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:51.689287 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:51.692090 (+ 2803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:51.692115 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:51.696501 (+ 4386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:51.696509 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:51.696780 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:51.696833 (+ 53us) client_negotiation.cc:769] Sending connection context
0810 00:45:51.696967 (+ 134us) client_negotiation.cc:240] Negotiation successful
0810 00:45:51.697148 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2977,"mutex_wait_us":30,"thread_start_us":1850,"threads_started":1}
I20260810 00:45:51.697942 7292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:51.685419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40000 (local address 127.28.6.61:37239)
0810 00:45:51.686125 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:51.686131 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:51.686155 (+ 24us) server_negotiation.cc:407] Connection header received
0810 00:45:51.686226 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:51.686230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:51.686299 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:51.687125 (+ 826us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:51.687439 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:51.689041 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:51.692401 (+ 3360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:51.696257 (+ 3856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:51.697280 (+ 1023us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:51.697410 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:51.697475 (+ 65us) server_negotiation.cc:299] Negotiation successful
0810 00:45:51.697602 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":360,"threads_started":1}
I20260810 00:45:51.697935 6496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "952b9af13aac44daae6473c2d7a19566" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" is_pre_election: true
I20260810 00:45:51.698315 6496 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 952b9af13aac44daae6473c2d7a19566 in term 1.
I20260810 00:45:51.698856 7285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:51.698302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.0.0.1:32772)
0810 00:45:51.698589 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:51.698682 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260810 00:45:51.698788 6212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "952b9af13aac44daae6473c2d7a19566" is_pre_election: true
I20260810 00:45:51.699141 6212 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8bd3d99c782483f87add2f7cb6f1a4c in term 1.
W20260810 00:45:51.699144 6458 leader_election.cc:334] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e6e4f5d8a3384fbcbee45036aad474fa (127.28.6.62:39675): Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
I20260810 00:45:51.699246 6177 leader_election.cc:302] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 952b9af13aac44daae6473c2d7a19566, e8bd3d99c782483f87add2f7cb6f1a4c; no voters:
I20260810 00:45:51.699415 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:51.698954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.0.0.1:32774)
0810 00:45:51.699160 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:51.699258 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260810 00:45:51.699646 7281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 00:45:51.699796 7281 raft_consensus.cc:482] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e6e4f5d8a3384fbcbee45036aad474fa)
I20260810 00:45:51.699797 6460 leader_election.cc:302] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 952b9af13aac44daae6473c2d7a19566, e8bd3d99c782483f87add2f7cb6f1a4c; no voters: e6e4f5d8a3384fbcbee45036aad474fa
I20260810 00:45:51.699900 7281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [term 1 FOLLOWER]: Advancing to term 2
I20260810 00:45:51.700235 7282 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 00:45:51.700416 7282 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [term 1 FOLLOWER]: Starting leader election (detected failure of leader e6e4f5d8a3384fbcbee45036aad474fa)
W20260810 00:45:51.700452 6177 leader_election.cc:334] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e6e4f5d8a3384fbcbee45036aad474fa (127.28.6.62:39675): Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
I20260810 00:45:51.700570 7282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [term 1 FOLLOWER]: Advancing to term 2
I20260810 00:45:51.703388 7281 raft_consensus.cc:504] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: false } } peers { permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 42769 } attrs { promote: false } }
I20260810 00:45:51.704200 7281 leader_election.cc:288] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [CANDIDATE]: Term 2 election: Requested vote from peers e6e4f5d8a3384fbcbee45036aad474fa (127.28.6.62:39675), e8bd3d99c782483f87add2f7cb6f1a4c (127.28.6.60:42769)
I20260810 00:45:51.704701 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:51.704284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.0.0.1:32780)
0810 00:45:51.704444 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:51.704563 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 00:45:51.704872 6496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "952b9af13aac44daae6473c2d7a19566" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c"
I20260810 00:45:51.704975 7282 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: false } } peers { permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 42769 } attrs { promote: false } }
I20260810 00:45:51.706054 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:51.705632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.0.0.1:32788)
0810 00:45:51.705799 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:51.705897 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 00:45:51.706212 7285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:51.705823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.0.0.1:32794)
0810 00:45:51.705973 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:51.706075 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 00:45:51.706360 6496 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 952b9af13aac44daae6473c2d7a19566 in current term 2: Already voted for candidate e8bd3d99c782483f87add2f7cb6f1a4c in this term.
I20260810 00:45:51.707396 7285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:51.706939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.0.0.1:32796)
0810 00:45:51.707098 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:51.707181 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260810 00:45:51.707683 7282 leader_election.cc:288] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [CANDIDATE]: Term 2 election: Requested vote from peers e6e4f5d8a3384fbcbee45036aad474fa (127.28.6.62:39675), 952b9af13aac44daae6473c2d7a19566 (127.28.6.61:37239)
W20260810 00:45:51.707737 6458 leader_election.cc:334] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e6e4f5d8a3384fbcbee45036aad474fa (127.28.6.62:39675): Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
W20260810 00:45:51.707989 6177 leader_election.cc:334] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e6e4f5d8a3384fbcbee45036aad474fa (127.28.6.62:39675): Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
I20260810 00:45:51.708441 6177 leader_election.cc:302] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 952b9af13aac44daae6473c2d7a19566; no voters: e6e4f5d8a3384fbcbee45036aad474fa, e8bd3d99c782483f87add2f7cb6f1a4c
I20260810 00:45:51.708758 7281 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 00:45:51.709820 6212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "952b9af13aac44daae6473c2d7a19566"
I20260810 00:45:51.710119 6212 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e8bd3d99c782483f87add2f7cb6f1a4c in current term 2: Already voted for candidate 952b9af13aac44daae6473c2d7a19566 in this term.
I20260810 00:45:51.710649 6460 leader_election.cc:302] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8bd3d99c782483f87add2f7cb6f1a4c; no voters: 952b9af13aac44daae6473c2d7a19566, e6e4f5d8a3384fbcbee45036aad474fa
I20260810 00:45:51.710974 7282 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 00:45:52.153604 7296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:52.152788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.28.6.1:53137)
0810 00:45:52.153331 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:52.153412 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":306,"threads_started":1}
I20260810 00:45:52.166519 7297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:52.165789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.28.6.3:50047)
0810 00:45:52.166272 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:52.166350 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":250,"threads_started":1}
I20260810 00:45:52.168315 7298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:52.167524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.28.6.2:55915)
0810 00:45:52.168021 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:52.168097 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":267,"threads_started":1}
I20260810 00:45:53.155296 7303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:53.154505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.28.6.1:52095)
0810 00:45:53.155035 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:53.155121 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":240,"threads_started":1}
I20260810 00:45:53.168246 7304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:53.167396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.28.6.3:39083)
0810 00:45:53.167941 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:53.168045 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":314,"threads_started":1}
I20260810 00:45:53.169849 7305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:53.169136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.28.6.2:60137)
0810 00:45:53.169609 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:53.169703 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":285,"threads_started":1}
I20260810 00:45:53.425088 7307 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:45:53.425290 7307 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: false } } peers { permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 42769 } attrs { promote: false } }
I20260810 00:45:53.425839 7307 leader_election.cc:288] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e6e4f5d8a3384fbcbee45036aad474fa (127.28.6.62:39675), 952b9af13aac44daae6473c2d7a19566 (127.28.6.61:37239)
I20260810 00:45:53.426481 6212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "952b9af13aac44daae6473c2d7a19566" is_pre_election: true
I20260810 00:45:53.426733 6212 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8bd3d99c782483f87add2f7cb6f1a4c in term 2.
I20260810 00:45:53.427161 7310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:53.426064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.0.0.1:33070)
0810 00:45:53.426727 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:53.426826 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":420,"threads_started":1}
I20260810 00:45:53.427289 6460 leader_election.cc:302] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 952b9af13aac44daae6473c2d7a19566, e8bd3d99c782483f87add2f7cb6f1a4c; no voters:
I20260810 00:45:53.427632 7307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 00:45:53.427783 7307 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 00:45:53.427912 7307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [term 2 FOLLOWER]: Advancing to term 3
I20260810 00:45:53.428673 7310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:53.428367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.0.0.1:33082)
0810 00:45:53.428497 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:53.428562 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 00:45:53.428900 6458 leader_election.cc:334] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e6e4f5d8a3384fbcbee45036aad474fa (127.28.6.62:39675): Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
I20260810 00:45:53.430385 7307 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: false } } peers { permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 42769 } attrs { promote: false } }
I20260810 00:45:53.431080 7307 leader_election.cc:288] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [CANDIDATE]: Term 3 election: Requested vote from peers e6e4f5d8a3384fbcbee45036aad474fa (127.28.6.62:39675), 952b9af13aac44daae6473c2d7a19566 (127.28.6.61:37239)
I20260810 00:45:53.431551 7310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:53.431080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.0.0.1:33096)
0810 00:45:53.431305 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:53.431386 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260810 00:45:53.431567 6212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "952b9af13aac44daae6473c2d7a19566"
I20260810 00:45:53.431763 6212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [term 2 FOLLOWER]: Advancing to term 3
I20260810 00:45:53.432562 7310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:53.432186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.0.0.1:33100)
0810 00:45:53.432357 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:53.432445 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260810 00:45:53.432744 6458 leader_election.cc:334] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer e6e4f5d8a3384fbcbee45036aad474fa (127.28.6.62:39675): Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
I20260810 00:45:53.434279 6212 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8bd3d99c782483f87add2f7cb6f1a4c in term 3.
I20260810 00:45:53.434695 6460 leader_election.cc:302] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 952b9af13aac44daae6473c2d7a19566, e8bd3d99c782483f87add2f7cb6f1a4c; no voters: e6e4f5d8a3384fbcbee45036aad474fa
I20260810 00:45:53.435016 7307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [term 3 FOLLOWER]: Leader election won for term 3
I20260810 00:45:53.435448 7307 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [term 3 LEADER]: Becoming Leader. State: Replica: e8bd3d99c782483f87add2f7cb6f1a4c, State: Running, Role: LEADER
I20260810 00:45:53.435839 7307 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: false } } peers { permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 42769 } attrs { promote: false } }
I20260810 00:45:53.437729 7314 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8bd3d99c782483f87add2f7cb6f1a4c. Latest consensus state: current_term: 3 leader_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: false } } peers { permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 42769 } attrs { promote: false } } }
I20260810 00:45:53.438086 7314 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [sys.catalog]: This master's current role is: LEADER
I20260810 00:45:53.438768 7317 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:45:53.442534 7317 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:45:53.443766 7317 catalog_manager.cc:1259] Loaded cluster ID: 55a845b735744815b615e8a0736d9f90
I20260810 00:45:53.443862 7317 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:45:53.445261 7317 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:45:53.446318 7317 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c: Loaded TSK: 0
I20260810 00:45:53.447394 7317 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 00:45:53.531570 6480 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43992:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 00:45:53.539036 6480 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 00:45:53.543286 7310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:53.542820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.0.0.1:33124)
0810 00:45:53.543058 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:53.543135 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260810 00:45:53.543386 6212 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [term 3 FOLLOWER]: Refusing update from remote peer e8bd3d99c782483f87add2f7cb6f1a4c: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260810 00:45:53.544176 7314 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260810 00:45:53.544401 7310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:53.544001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.0.0.1:33136)
0810 00:45:53.544184 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:53.544264 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 00:45:53.544602 6458 consensus_peers.cc:487] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c -> Peer e6e4f5d8a3384fbcbee45036aad474fa (127.28.6.62:39675): Couldn't send request to peer e6e4f5d8a3384fbcbee45036aad474fa. Status: Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 00:45:53.547596 7319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8bd3d99c782483f87add2f7cb6f1a4c. Latest consensus state: current_term: 3 leader_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: false } } peers { permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 42769 } attrs { promote: false } } }
I20260810 00:45:53.547904 7319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 00:45:53.548987 7307 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: false } } peers { permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 42769 } attrs { promote: false } } }
I20260810 00:45:53.549429 7307 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c [sys.catalog]: This master's current role is: LEADER
I20260810 00:45:53.549688 7319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "e6e4f5d8a3384fbcbee45036aad474fa" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39675 } } peers { permanent_uuid: "952b9af13aac44daae6473c2d7a19566" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 37239 } attrs { promote: false } } peers { permanent_uuid: "e8bd3d99c782483f87add2f7cb6f1a4c" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 42769 } attrs { promote: false } } }
I20260810 00:45:53.549926 7319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 952b9af13aac44daae6473c2d7a19566 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 00:45:53.570639 7310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:53.557840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:39253 (local address 127.0.0.1:35242)
0810 00:45:53.557975 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:53.557998 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:53.558117 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:53.559095 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:53.559101 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:53.559130 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:53.559527 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:53.559546 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:53.562988 (+ 3442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:53.562997 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:53.565892 (+ 2895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:53.565913 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:53.569775 (+ 3862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:53.569784 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:53.570000 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:53.570037 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 00:45:53.570111 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 00:45:53.570266 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 00:45:53.573200 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:53.558066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35242 (local address 127.28.6.3:39253)
0810 00:45:53.558595 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:53.558601 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:53.558620 (+ 19us) server_negotiation.cc:407] Connection header received
0810 00:45:53.558777 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:53.558782 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:53.558923 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:53.559718 (+ 795us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:53.559981 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:53.562748 (+ 2767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:53.567490 (+ 4742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:53.569510 (+ 2020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:53.572568 (+ 3058us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:53.572676 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:53.572727 (+ 51us) server_negotiation.cc:299] Negotiation successful
0810 00:45:53.572869 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":269,"threads_started":1}
I20260810 00:45:53.574247 7073 tablet_service.cc:1467] Processing CreateTablet for tablet 3955f018e6fb450c888dc8f2a42af3b6 (DEFAULT_TABLE table=test-table [id=adc2b9965a7e41749349b40a65a56891]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:53.574749 7073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3955f018e6fb450c888dc8f2a42af3b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:53.580438 7335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:53.559076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39134 (local address 127.28.6.2:39127)
0810 00:45:53.570114 (+ 11038us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:53.570120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:53.570927 (+ 807us) server_negotiation.cc:407] Connection header received
0810 00:45:53.571009 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:53.571014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:53.571094 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:53.571196 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:53.572066 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:53.573115 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:53.578080 (+ 4965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:53.579376 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:53.579519 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:53.579623 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:53.579950 (+ 327us) server_negotiation.cc:299] Negotiation successful
0810 00:45:53.580095 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10903,"thread_start_us":11028,"threads_started":1}
I20260810 00:45:53.580475 7336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:53.558892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:39127 (local address 127.0.0.1:39134)
0810 00:45:53.570649 (+ 11757us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:53.570669 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:53.570792 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:53.571437 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:53.571442 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:53.571465 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:53.571868 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:53.571881 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:53.573292 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:53.573301 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:53.577856 (+ 4555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:53.577876 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:53.579576 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:53.579583 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:53.579759 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:53.579797 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 00:45:53.579903 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 00:45:53.580045 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11616,"mutex_wait_us":51,"thread_start_us":17979,"threads_started":1}
I20260810 00:45:53.580956 7310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:53.560496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:42679 (local address 127.0.0.1:44952)
0810 00:45:53.570860 (+ 10364us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:53.570878 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:53.572399 (+ 1521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:53.574360 (+ 1961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:53.574364 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:53.574389 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:53.574549 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:53.574562 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:53.577209 (+ 2647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:53.577214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:53.578736 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:53.578748 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:53.580118 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:53.580127 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:53.580350 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:53.580384 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 00:45:53.580467 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 00:45:53.580592 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10263,"mutex_wait_us":52,"thread_start_us":9862,"threads_started":1}
I20260810 00:45:53.581456 6860 tablet_service.cc:1467] Processing CreateTablet for tablet 3955f018e6fb450c888dc8f2a42af3b6 (DEFAULT_TABLE table=test-table [id=adc2b9965a7e41749349b40a65a56891]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:53.581833 7341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:53.560949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44952 (local address 127.28.6.1:42679)
0810 00:45:53.574013 (+ 13064us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:53.574020 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:53.574038 (+ 18us) server_negotiation.cc:407] Connection header received
0810 00:45:53.574105 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:53.574110 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:53.574200 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:53.574757 (+ 557us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:53.575048 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:53.576087 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:53.578919 (+ 2832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:53.579954 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:53.581303 (+ 1349us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:53.581387 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:53.581428 (+ 41us) server_negotiation.cc:299] Negotiation successful
0810 00:45:53.581519 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12924,"thread_start_us":12799,"threads_started":1}
I20260810 00:45:53.581895 6860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3955f018e6fb450c888dc8f2a42af3b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:53.582527 6641 tablet_service.cc:1467] Processing CreateTablet for tablet 3955f018e6fb450c888dc8f2a42af3b6 (DEFAULT_TABLE table=test-table [id=adc2b9965a7e41749349b40a65a56891]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:53.583048 6641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3955f018e6fb450c888dc8f2a42af3b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:53.586987 7342 tablet_bootstrap.cc:492] T 3955f018e6fb450c888dc8f2a42af3b6 P 672dbbeddd0a4ecdb51a8f256074d9b2: Bootstrap starting.
I20260810 00:45:53.587498 7343 tablet_bootstrap.cc:492] T 3955f018e6fb450c888dc8f2a42af3b6 P dc1d57df533e4f8a87d5f4595ec26f31: Bootstrap starting.
I20260810 00:45:53.588425 7344 tablet_bootstrap.cc:492] T 3955f018e6fb450c888dc8f2a42af3b6 P 6668f2f5a228466b8d3b7b8de4661e3d: Bootstrap starting.
I20260810 00:45:53.589092 7342 tablet_bootstrap.cc:654] T 3955f018e6fb450c888dc8f2a42af3b6 P 672dbbeddd0a4ecdb51a8f256074d9b2: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:53.589495 7343 tablet_bootstrap.cc:654] T 3955f018e6fb450c888dc8f2a42af3b6 P dc1d57df533e4f8a87d5f4595ec26f31: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:53.590325 7344 tablet_bootstrap.cc:654] T 3955f018e6fb450c888dc8f2a42af3b6 P 6668f2f5a228466b8d3b7b8de4661e3d: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:53.593222 7344 tablet_bootstrap.cc:492] T 3955f018e6fb450c888dc8f2a42af3b6 P 6668f2f5a228466b8d3b7b8de4661e3d: No bootstrap required, opened a new log
I20260810 00:45:53.593408 7344 ts_tablet_manager.cc:1397] T 3955f018e6fb450c888dc8f2a42af3b6 P 6668f2f5a228466b8d3b7b8de4661e3d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260810 00:45:53.594316 7344 raft_consensus.cc:348] T 3955f018e6fb450c888dc8f2a42af3b6 P 6668f2f5a228466b8d3b7b8de4661e3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc1d57df533e4f8a87d5f4595ec26f31" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39253 } } peers { permanent_uuid: "672dbbeddd0a4ecdb51a8f256074d9b2" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 39127 } } peers { permanent_uuid: "6668f2f5a228466b8d3b7b8de4661e3d" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 42679 } }
I20260810 00:45:53.594575 7344 raft_consensus.cc:374] T 3955f018e6fb450c888dc8f2a42af3b6 P 6668f2f5a228466b8d3b7b8de4661e3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:53.594676 7344 raft_consensus.cc:729] T 3955f018e6fb450c888dc8f2a42af3b6 P 6668f2f5a228466b8d3b7b8de4661e3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6668f2f5a228466b8d3b7b8de4661e3d, State: Initialized, Role: FOLLOWER
I20260810 00:45:53.594964 7342 tablet_bootstrap.cc:492] T 3955f018e6fb450c888dc8f2a42af3b6 P 672dbbeddd0a4ecdb51a8f256074d9b2: No bootstrap required, opened a new log
I20260810 00:45:53.595119 7342 ts_tablet_manager.cc:1397] T 3955f018e6fb450c888dc8f2a42af3b6 P 672dbbeddd0a4ecdb51a8f256074d9b2: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260810 00:45:53.594991 7344 consensus_queue.cc:260] T 3955f018e6fb450c888dc8f2a42af3b6 P 6668f2f5a228466b8d3b7b8de4661e3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc1d57df533e4f8a87d5f4595ec26f31" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39253 } } peers { permanent_uuid: "672dbbeddd0a4ecdb51a8f256074d9b2" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 39127 } } peers { permanent_uuid: "6668f2f5a228466b8d3b7b8de4661e3d" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 42679 } }
I20260810 00:45:53.595237 7343 tablet_bootstrap.cc:492] T 3955f018e6fb450c888dc8f2a42af3b6 P dc1d57df533e4f8a87d5f4595ec26f31: No bootstrap required, opened a new log
I20260810 00:45:53.595382 7343 ts_tablet_manager.cc:1397] T 3955f018e6fb450c888dc8f2a42af3b6 P dc1d57df533e4f8a87d5f4595ec26f31: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260810 00:45:53.595995 7344 ts_tablet_manager.cc:1428] T 3955f018e6fb450c888dc8f2a42af3b6 P 6668f2f5a228466b8d3b7b8de4661e3d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 00:45:53.595963 7342 raft_consensus.cc:348] T 3955f018e6fb450c888dc8f2a42af3b6 P 672dbbeddd0a4ecdb51a8f256074d9b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc1d57df533e4f8a87d5f4595ec26f31" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39253 } } peers { permanent_uuid: "672dbbeddd0a4ecdb51a8f256074d9b2" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 39127 } } peers { permanent_uuid: "6668f2f5a228466b8d3b7b8de4661e3d" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 42679 } }
I20260810 00:45:53.596227 7342 raft_consensus.cc:374] T 3955f018e6fb450c888dc8f2a42af3b6 P 672dbbeddd0a4ecdb51a8f256074d9b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:53.596320 7342 raft_consensus.cc:729] T 3955f018e6fb450c888dc8f2a42af3b6 P 672dbbeddd0a4ecdb51a8f256074d9b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 672dbbeddd0a4ecdb51a8f256074d9b2, State: Initialized, Role: FOLLOWER
I20260810 00:45:53.596210 7343 raft_consensus.cc:348] T 3955f018e6fb450c888dc8f2a42af3b6 P dc1d57df533e4f8a87d5f4595ec26f31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc1d57df533e4f8a87d5f4595ec26f31" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39253 } } peers { permanent_uuid: "672dbbeddd0a4ecdb51a8f256074d9b2" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 39127 } } peers { permanent_uuid: "6668f2f5a228466b8d3b7b8de4661e3d" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 42679 } }
I20260810 00:45:53.596498 7343 raft_consensus.cc:374] T 3955f018e6fb450c888dc8f2a42af3b6 P dc1d57df533e4f8a87d5f4595ec26f31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:53.596642 7343 raft_consensus.cc:729] T 3955f018e6fb450c888dc8f2a42af3b6 P dc1d57df533e4f8a87d5f4595ec26f31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc1d57df533e4f8a87d5f4595ec26f31, State: Initialized, Role: FOLLOWER
I20260810 00:45:53.596555 7342 consensus_queue.cc:260] T 3955f018e6fb450c888dc8f2a42af3b6 P 672dbbeddd0a4ecdb51a8f256074d9b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc1d57df533e4f8a87d5f4595ec26f31" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39253 } } peers { permanent_uuid: "672dbbeddd0a4ecdb51a8f256074d9b2" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 39127 } } peers { permanent_uuid: "6668f2f5a228466b8d3b7b8de4661e3d" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 42679 } }
I20260810 00:45:53.596907 7343 consensus_queue.cc:260] T 3955f018e6fb450c888dc8f2a42af3b6 P dc1d57df533e4f8a87d5f4595ec26f31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc1d57df533e4f8a87d5f4595ec26f31" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39253 } } peers { permanent_uuid: "672dbbeddd0a4ecdb51a8f256074d9b2" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 39127 } } peers { permanent_uuid: "6668f2f5a228466b8d3b7b8de4661e3d" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 42679 } }
I20260810 00:45:53.597301 7303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:53.596582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.28.6.1:40593)
0810 00:45:53.596996 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:53.597101 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303}
I20260810 00:45:53.600201 7303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:53.599708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.28.6.1:53409)
0810 00:45:53.599931 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:53.600026 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260810 00:45:53.600207 7342 ts_tablet_manager.cc:1428] T 3955f018e6fb450c888dc8f2a42af3b6 P 672dbbeddd0a4ecdb51a8f256074d9b2: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260810 00:45:53.600418 7343 ts_tablet_manager.cc:1428] T 3955f018e6fb450c888dc8f2a42af3b6 P dc1d57df533e4f8a87d5f4595ec26f31: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260810 00:45:53.601341 6739 heartbeater.cc:503] Master 127.28.6.60:42769 was elected leader, sending a full tablet report...
I20260810 00:45:53.601625 7304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:53.601043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.28.6.3:49509)
0810 00:45:53.601399 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:53.601488 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234}
I20260810 00:45:53.603672 7169 heartbeater.cc:503] Master 127.28.6.60:42769 was elected leader, sending a full tablet report...
I20260810 00:45:53.604730 7305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:53.604033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.28.6.2:45333)
0810 00:45:53.604393 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:53.604491 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241}
I20260810 00:45:53.606057 7304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:53.605480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.28.6.3:55515)
0810 00:45:53.605789 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:53.605875 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216}
I20260810 00:45:53.606379 6955 heartbeater.cc:503] Master 127.28.6.60:42769 was elected leader, sending a full tablet report...
I20260810 00:45:53.607095 7305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:53.605945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.28.6.2:33125)
0810 00:45:53.606834 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:53.606937 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":767}
I20260810 00:45:53.704545 7348 raft_consensus.cc:482] T 3955f018e6fb450c888dc8f2a42af3b6 P 6668f2f5a228466b8d3b7b8de4661e3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:45:53.704831 7348 raft_consensus.cc:504] T 3955f018e6fb450c888dc8f2a42af3b6 P 6668f2f5a228466b8d3b7b8de4661e3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc1d57df533e4f8a87d5f4595ec26f31" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39253 } } peers { permanent_uuid: "672dbbeddd0a4ecdb51a8f256074d9b2" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 39127 } } peers { permanent_uuid: "6668f2f5a228466b8d3b7b8de4661e3d" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 42679 } }
I20260810 00:45:53.706413 7348 leader_election.cc:288] T 3955f018e6fb450c888dc8f2a42af3b6 P 6668f2f5a228466b8d3b7b8de4661e3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc1d57df533e4f8a87d5f4595ec26f31 (127.28.6.3:39253), 672dbbeddd0a4ecdb51a8f256074d9b2 (127.28.6.2:39127)
I20260810 00:45:53.713779 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:53.706697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:52979 (local address 127.28.6.3:39253)
0810 00:45:53.706942 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:53.706950 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:53.706973 (+ 23us) server_negotiation.cc:407] Connection header received
0810 00:45:53.707226 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:53.707236 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:53.707330 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:53.707451 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:53.708075 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:53.709056 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:53.711684 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:53.712890 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:53.713026 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:53.713133 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:53.713436 (+ 303us) server_negotiation.cc:299] Negotiation successful
0810 00:45:53.713531 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 00:45:53.713994 7303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:53.706670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:39253 (local address 127.28.6.1:52979)
0810 00:45:53.706844 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:53.706865 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:53.707037 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:53.707496 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:53.707501 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:53.707532 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:53.707912 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:53.707923 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:53.709210 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:53.709215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:53.711533 (+ 2318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:53.711548 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:53.713079 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:53.713084 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:53.713246 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:53.713295 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 00:45:53.713402 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 00:45:53.713555 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 00:45:53.714107 7335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:53.706875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:38957 (local address 127.28.6.2:39127)
0810 00:45:53.707136 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:53.707142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:53.707697 (+ 555us) server_negotiation.cc:407] Connection header received
0810 00:45:53.707907 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:53.707912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:53.707980 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:53.708097 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:53.708592 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:53.709496 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:53.711921 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:53.713135 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:53.713247 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:53.713315 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:53.713715 (+ 400us) server_negotiation.cc:299] Negotiation successful
0810 00:45:53.713866 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260810 00:45:53.714242 7352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:53.706668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:39127 (local address 127.28.6.1:38957)
0810 00:45:53.707574 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:53.707599 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:53.707731 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:53.708122 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:53.708125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:53.708147 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:53.708436 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:53.708443 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:53.709643 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:53.709648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:53.711776 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:53.711787 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:53.713343 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:53.713351 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:53.713548 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:53.713577 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 00:45:53.713668 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 00:45:53.713788 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"mutex_wait_us":193,"thread_start_us":334,"threads_started":1}
I20260810 00:45:53.715056 6884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3955f018e6fb450c888dc8f2a42af3b6" candidate_uuid: "6668f2f5a228466b8d3b7b8de4661e3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "672dbbeddd0a4ecdb51a8f256074d9b2" is_pre_election: true
I20260810 00:45:53.715109 7103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3955f018e6fb450c888dc8f2a42af3b6" candidate_uuid: "6668f2f5a228466b8d3b7b8de4661e3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc1d57df533e4f8a87d5f4595ec26f31" is_pre_election: true
I20260810 00:45:53.715317 6884 raft_consensus.cc:2454] T 3955f018e6fb450c888dc8f2a42af3b6 P 672dbbeddd0a4ecdb51a8f256074d9b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6668f2f5a228466b8d3b7b8de4661e3d in term 0.
I20260810 00:45:53.715555 7103 raft_consensus.cc:2454] T 3955f018e6fb450c888dc8f2a42af3b6 P dc1d57df533e4f8a87d5f4595ec26f31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6668f2f5a228466b8d3b7b8de4661e3d in term 0.
I20260810 00:45:53.715852 6574 leader_election.cc:302] T 3955f018e6fb450c888dc8f2a42af3b6 P 6668f2f5a228466b8d3b7b8de4661e3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6668f2f5a228466b8d3b7b8de4661e3d, 672dbbeddd0a4ecdb51a8f256074d9b2; no voters:
I20260810 00:45:53.716235 7348 raft_consensus.cc:2781] T 3955f018e6fb450c888dc8f2a42af3b6 P 6668f2f5a228466b8d3b7b8de4661e3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 00:45:53.716374 7348 raft_consensus.cc:482] T 3955f018e6fb450c888dc8f2a42af3b6 P 6668f2f5a228466b8d3b7b8de4661e3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 00:45:53.716480 7348 raft_consensus.cc:3037] T 3955f018e6fb450c888dc8f2a42af3b6 P 6668f2f5a228466b8d3b7b8de4661e3d [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:53.718554 7348 raft_consensus.cc:504] T 3955f018e6fb450c888dc8f2a42af3b6 P 6668f2f5a228466b8d3b7b8de4661e3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc1d57df533e4f8a87d5f4595ec26f31" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39253 } } peers { permanent_uuid: "672dbbeddd0a4ecdb51a8f256074d9b2" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 39127 } } peers { permanent_uuid: "6668f2f5a228466b8d3b7b8de4661e3d" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 42679 } }
I20260810 00:45:53.719244 7348 leader_election.cc:288] T 3955f018e6fb450c888dc8f2a42af3b6 P 6668f2f5a228466b8d3b7b8de4661e3d [CANDIDATE]: Term 1 election: Requested vote from peers dc1d57df533e4f8a87d5f4595ec26f31 (127.28.6.3:39253), 672dbbeddd0a4ecdb51a8f256074d9b2 (127.28.6.2:39127)
I20260810 00:45:53.719565 7103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3955f018e6fb450c888dc8f2a42af3b6" candidate_uuid: "6668f2f5a228466b8d3b7b8de4661e3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc1d57df533e4f8a87d5f4595ec26f31"
W20260810 00:45:53.719647 6744 tablet.cc:2369] T 3955f018e6fb450c888dc8f2a42af3b6 P 6668f2f5a228466b8d3b7b8de4661e3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:45:53.719794 7103 raft_consensus.cc:3037] T 3955f018e6fb450c888dc8f2a42af3b6 P dc1d57df533e4f8a87d5f4595ec26f31 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:53.719858 6884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3955f018e6fb450c888dc8f2a42af3b6" candidate_uuid: "6668f2f5a228466b8d3b7b8de4661e3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "672dbbeddd0a4ecdb51a8f256074d9b2"
I20260810 00:45:53.720083 6884 raft_consensus.cc:3037] T 3955f018e6fb450c888dc8f2a42af3b6 P 672dbbeddd0a4ecdb51a8f256074d9b2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:53.723047 6884 raft_consensus.cc:2454] T 3955f018e6fb450c888dc8f2a42af3b6 P 672dbbeddd0a4ecdb51a8f256074d9b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6668f2f5a228466b8d3b7b8de4661e3d in term 1.
I20260810 00:45:53.723047 7103 raft_consensus.cc:2454] T 3955f018e6fb450c888dc8f2a42af3b6 P dc1d57df533e4f8a87d5f4595ec26f31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6668f2f5a228466b8d3b7b8de4661e3d in term 1.
I20260810 00:45:53.723445 6570 leader_election.cc:302] T 3955f018e6fb450c888dc8f2a42af3b6 P 6668f2f5a228466b8d3b7b8de4661e3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6668f2f5a228466b8d3b7b8de4661e3d, dc1d57df533e4f8a87d5f4595ec26f31; no voters:
I20260810 00:45:53.723778 7348 raft_consensus.cc:2781] T 3955f018e6fb450c888dc8f2a42af3b6 P 6668f2f5a228466b8d3b7b8de4661e3d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:45:53.724257 7348 raft_consensus.cc:686] T 3955f018e6fb450c888dc8f2a42af3b6 P 6668f2f5a228466b8d3b7b8de4661e3d [term 1 LEADER]: Becoming Leader. State: Replica: 6668f2f5a228466b8d3b7b8de4661e3d, State: Running, Role: LEADER
I20260810 00:45:53.724633 7348 consensus_queue.cc:237] T 3955f018e6fb450c888dc8f2a42af3b6 P 6668f2f5a228466b8d3b7b8de4661e3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc1d57df533e4f8a87d5f4595ec26f31" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39253 } } peers { permanent_uuid: "672dbbeddd0a4ecdb51a8f256074d9b2" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 39127 } } peers { permanent_uuid: "6668f2f5a228466b8d3b7b8de4661e3d" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 42679 } }
I20260810 00:45:53.725605 7303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:53.725230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.28.6.1:38595)
0810 00:45:53.725400 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:53.725471 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 00:45:53.728638 6480 catalog_manager.cc:5495] T 3955f018e6fb450c888dc8f2a42af3b6 P 6668f2f5a228466b8d3b7b8de4661e3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6668f2f5a228466b8d3b7b8de4661e3d (127.28.6.1). New cstate: current_term: 1 leader_uuid: "6668f2f5a228466b8d3b7b8de4661e3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc1d57df533e4f8a87d5f4595ec26f31" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "672dbbeddd0a4ecdb51a8f256074d9b2" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 39127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6668f2f5a228466b8d3b7b8de4661e3d" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 42679 } health_report { overall_health: HEALTHY } } }
I20260810 00:45:53.758385 7341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:53.753605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44954 (local address 127.28.6.1:42679)
0810 00:45:53.753778 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:53.753782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:53.753799 (+ 17us) server_negotiation.cc:407] Connection header received
0810 00:45:53.753970 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:53.753974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:53.754046 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:53.754167 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:45:53.754484 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:53.755235 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:53.756515 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:53.756934 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:53.757021 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:53.757380 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:45:53.757904 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:45:53.758005 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:53.758116 (+ 111us) server_negotiation.cc:299] Negotiation successful
0810 00:45:53.758177 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 00:45:53.762137 6884 raft_consensus.cc:1261] T 3955f018e6fb450c888dc8f2a42af3b6 P 672dbbeddd0a4ecdb51a8f256074d9b2 [term 1 FOLLOWER]: Refusing update from remote peer 6668f2f5a228466b8d3b7b8de4661e3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:45:53.762596 7103 raft_consensus.cc:1261] T 3955f018e6fb450c888dc8f2a42af3b6 P dc1d57df533e4f8a87d5f4595ec26f31 [term 1 FOLLOWER]: Refusing update from remote peer 6668f2f5a228466b8d3b7b8de4661e3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:45:53.762990 7354 consensus_queue.cc:1035] T 3955f018e6fb450c888dc8f2a42af3b6 P 6668f2f5a228466b8d3b7b8de4661e3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "672dbbeddd0a4ecdb51a8f256074d9b2" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 39127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:45:53.763432 7348 consensus_queue.cc:1035] T 3955f018e6fb450c888dc8f2a42af3b6 P 6668f2f5a228466b8d3b7b8de4661e3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc1d57df533e4f8a87d5f4595ec26f31" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:45:53.765875 7305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:53.765319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.28.6.2:51603)
0810 00:45:53.765515 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:53.765641 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260810 00:45:53.766557 7304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:53.765874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.28.6.3:38763)
0810 00:45:53.766169 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:53.766317 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260810 00:45:53.782378 7304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:53.781928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.28.6.3:46461)
0810 00:45:53.782126 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:53.782203 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260810 00:45:53.790480 7303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:53.790059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.28.6.1:38817)
0810 00:45:53.790239 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:53.790337 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260810 00:45:53.796686 7303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:53.796205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.28.6.1:50613)
0810 00:45:53.796386 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:53.796499 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260810 00:45:53.799978 7305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:53.799440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.28.6.2:36793)
0810 00:45:53.799691 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:53.799781 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260810 00:45:53.807502 6480 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:45:53.807746 6480 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:45:53.813212 6480 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e8bd3d99c782483f87add2f7cb6f1a4c: Sending DeleteTablet for 3 replicas of tablet 3955f018e6fb450c888dc8f2a42af3b6
I20260810 00:45:53.814611 7073 tablet_service.cc:1514] Processing DeleteTablet for tablet 3955f018e6fb450c888dc8f2a42af3b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:45:53 UTC) from {username='oryx', principal='oryx/127.28.6.60@KRBTEST.COM'} at 127.0.0.1:35242
I20260810 00:45:53.815054 6641 tablet_service.cc:1514] Processing DeleteTablet for tablet 3955f018e6fb450c888dc8f2a42af3b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:45:53 UTC) from {username='oryx', principal='oryx/127.28.6.60@KRBTEST.COM'} at 127.0.0.1:44952
I20260810 00:45:53.815193 6860 tablet_service.cc:1514] Processing DeleteTablet for tablet 3955f018e6fb450c888dc8f2a42af3b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:45:53 UTC) from {username='oryx', principal='oryx/127.28.6.60@KRBTEST.COM'} at 127.0.0.1:39134
I20260810 00:45:53.815832 7376 tablet_replica.cc:330] T 3955f018e6fb450c888dc8f2a42af3b6 P 6668f2f5a228466b8d3b7b8de4661e3d: stopping tablet replica
I20260810 00:45:53.816082 7376 raft_consensus.cc:2229] T 3955f018e6fb450c888dc8f2a42af3b6 P 6668f2f5a228466b8d3b7b8de4661e3d [term 1 LEADER]: Raft consensus shutting down.
I20260810 00:45:53.816728 7376 raft_consensus.cc:2258] T 3955f018e6fb450c888dc8f2a42af3b6 P 6668f2f5a228466b8d3b7b8de4661e3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:45:53.818821 7376 ts_tablet_manager.cc:1905] T 3955f018e6fb450c888dc8f2a42af3b6 P 6668f2f5a228466b8d3b7b8de4661e3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:45:53.822468 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 6447
I20260810 00:45:53.826484 7374 tablet_replica.cc:330] T 3955f018e6fb450c888dc8f2a42af3b6 P dc1d57df533e4f8a87d5f4595ec26f31: stopping tablet replica
I20260810 00:45:53.826743 7374 raft_consensus.cc:2229] T 3955f018e6fb450c888dc8f2a42af3b6 P dc1d57df533e4f8a87d5f4595ec26f31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:45:53.827131 7374 raft_consensus.cc:2258] T 3955f018e6fb450c888dc8f2a42af3b6 P dc1d57df533e4f8a87d5f4595ec26f31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:45:53.828374 7374 ts_tablet_manager.cc:1905] T 3955f018e6fb450c888dc8f2a42af3b6 P dc1d57df533e4f8a87d5f4595ec26f31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:45:53.831348 7378 tablet_replica.cc:330] T 3955f018e6fb450c888dc8f2a42af3b6 P 672dbbeddd0a4ecdb51a8f256074d9b2: stopping tablet replica
I20260810 00:45:53.831557 7378 raft_consensus.cc:2229] T 3955f018e6fb450c888dc8f2a42af3b6 P 672dbbeddd0a4ecdb51a8f256074d9b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:45:53.833241 7376 ts_tablet_manager.cc:1918] T 3955f018e6fb450c888dc8f2a42af3b6 P 6668f2f5a228466b8d3b7b8de4661e3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:45:53.833460 7376 log.cc:1196] T 3955f018e6fb450c888dc8f2a42af3b6 P 6668f2f5a228466b8d3b7b8de4661e3d: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-0/wal/wals/3955f018e6fb450c888dc8f2a42af3b6
I20260810 00:45:53.834085 7376 ts_tablet_manager.cc:1939] T 3955f018e6fb450c888dc8f2a42af3b6 P 6668f2f5a228466b8d3b7b8de4661e3d: Deleting consensus metadata
I20260810 00:45:53.835482 6458 catalog_manager.cc:4841] TS 6668f2f5a228466b8d3b7b8de4661e3d (127.28.6.1:42679): tablet 3955f018e6fb450c888dc8f2a42af3b6 (table test-table [id=adc2b9965a7e41749349b40a65a56891]) successfully deleted
I20260810 00:45:53.836750 7374 ts_tablet_manager.cc:1918] T 3955f018e6fb450c888dc8f2a42af3b6 P dc1d57df533e4f8a87d5f4595ec26f31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:45:53.836887 7374 log.cc:1196] T 3955f018e6fb450c888dc8f2a42af3b6 P dc1d57df533e4f8a87d5f4595ec26f31: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-2/wal/wals/3955f018e6fb450c888dc8f2a42af3b6
I20260810 00:45:53.837430 7374 ts_tablet_manager.cc:1939] T 3955f018e6fb450c888dc8f2a42af3b6 P dc1d57df533e4f8a87d5f4595ec26f31: Deleting consensus metadata
I20260810 00:45:53.831903 7378 raft_consensus.cc:2258] T 3955f018e6fb450c888dc8f2a42af3b6 P 672dbbeddd0a4ecdb51a8f256074d9b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:45:53.838400 6456 catalog_manager.cc:4841] TS dc1d57df533e4f8a87d5f4595ec26f31 (127.28.6.3:39253): tablet 3955f018e6fb450c888dc8f2a42af3b6 (table test-table [id=adc2b9965a7e41749349b40a65a56891]) successfully deleted
I20260810 00:45:53.839418 7378 ts_tablet_manager.cc:1905] T 3955f018e6fb450c888dc8f2a42af3b6 P 672dbbeddd0a4ecdb51a8f256074d9b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:45:53.845541 7378 ts_tablet_manager.cc:1918] T 3955f018e6fb450c888dc8f2a42af3b6 P 672dbbeddd0a4ecdb51a8f256074d9b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:45:53.845718 7378 log.cc:1196] T 3955f018e6fb450c888dc8f2a42af3b6 P 672dbbeddd0a4ecdb51a8f256074d9b2: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786322687226232-28696-0/minicluster-data/ts-1/wal/wals/3955f018e6fb450c888dc8f2a42af3b6
I20260810 00:45:53.846212 7378 ts_tablet_manager.cc:1939] T 3955f018e6fb450c888dc8f2a42af3b6 P 672dbbeddd0a4ecdb51a8f256074d9b2: Deleting consensus metadata
I20260810 00:45:53.847348 6460 catalog_manager.cc:4841] TS 672dbbeddd0a4ecdb51a8f256074d9b2 (127.28.6.2:39127): tablet 3955f018e6fb450c888dc8f2a42af3b6 (table test-table [id=adc2b9965a7e41749349b40a65a56891]) successfully deleted
I20260810 00:45:53.849788 7341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:53.824127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44962 (local address 127.28.6.1:42679)
0810 00:45:53.824342 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:53.824352 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:53.831592 (+ 7240us) server_negotiation.cc:407] Connection header received
0810 00:45:53.831891 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:53.831898 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:53.831985 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:53.832081 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:53.834160 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:53.835509 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:53.837354 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:53.838056 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:53.838191 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:53.842047 (+ 3856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:53.842062 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:53.842082 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:53.842140 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:53.846535 (+ 4395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:53.847656 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:53.847663 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:53.847666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:53.847836 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:53.848219 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:53.848222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:53.848224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:53.848636 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:53.848912 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:53.849242 (+ 330us) server_negotiation.cc:299] Negotiation successful
0810 00:45:53.849394 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 00:45:53.850575 6728 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:45:53.952710 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 6447
I20260810 00:45:53.982733 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 6755
I20260810 00:45:53.996685 7335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:53.983534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39136 (local address 127.28.6.2:39127)
0810 00:45:53.983741 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:53.983749 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:53.983834 (+ 85us) server_negotiation.cc:407] Connection header received
0810 00:45:53.984068 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:53.984073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:53.984143 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:53.984276 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:53.985602 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:53.986348 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:53.987731 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:53.988136 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:53.988328 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:53.991317 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:53.991332 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:53.991350 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:53.991396 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:53.994123 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:53.994739 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:53.994746 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:53.994748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:53.994876 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:53.995411 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:53.995414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:53.995416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:53.995793 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:53.996031 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:53.996260 (+ 229us) server_negotiation.cc:299] Negotiation successful
0810 00:45:53.996419 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 00:45:53.997416 6946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:45:54.101845 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 6755
I20260810 00:45:54.109701 7384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:54.108863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.0.0.1:33148)
0810 00:45:54.109460 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:54.109542 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":365,"threads_started":1}
I20260810 00:45:54.111418 7384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:54.110989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39675 (local address 127.0.0.1:33160)
0810 00:45:54.111170 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:54.111256 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:39675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260810 00:45:54.134860 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 6972
I20260810 00:45:54.149572 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:54.135738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35248 (local address 127.28.6.3:39253)
0810 00:45:54.135929 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:54.135933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:54.135951 (+ 18us) server_negotiation.cc:407] Connection header received
0810 00:45:54.136080 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:54.136083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:54.136147 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:54.136245 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:54.138107 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:54.138845 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:54.140171 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:54.140583 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:54.140732 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:54.143779 (+ 3047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:54.143801 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:54.143825 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:54.143893 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:54.147165 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:54.147752 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:54.147759 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:54.147761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:54.147883 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:54.148213 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:54.148216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:54.148218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:54.148578 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:54.148802 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:54.149041 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 00:45:54.149197 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 00:45:54.150291 7160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:45:54.269136 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 6972
I20260810 00:45:54.301366 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 5999
I20260810 00:45:54.302529 6248 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:45:54.420454 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 5999
I20260810 00:45:54.451064 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 6140
I20260810 00:45:54.452162 6529 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:45:54.574473 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 6140
2026-08-10T00:45:54Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (12067 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:45:54 dist-test-slave-1nl6 krb5kdc[7401](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:45:54 dist-test-slave-1nl6 krb5kdc[7401](info): set up 2 sockets
Aug 10 00:45:54 dist-test-slave-1nl6 krb5kdc[7401](info): commencing operation
krb5kdc: starting...
W20260810 00:45:55.907106 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.271s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:45:56 dist-test-slave-1nl6 krb5kdc[7401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:45:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:45:56Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.6.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.6.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.6.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.62.keytab.
Entry for principal oryx/127.28.6.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.62.keytab.
I20260810 00:45:56.140734 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:45123
--webserver_interface=127.28.6.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.62.keytab
--principal=oryx/127.28.6.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:43023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:45123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:45:56.470204 7423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:56.470518 7423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:56.470688 7423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:56.477686 7423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:45:56.477821 7423 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 00:45:56.477923 7423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:56.478040 7423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:56.478133 7423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:45:56.478230 7423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:45:56.487563 7423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:43023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:45123
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.62.keytab
--principal=oryx/127.28.6.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:45123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:56.488754 7423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:56.490074 7423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:56.501801 7429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:56.501847 7431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:56.502025 7433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:56.502497 7423 server_base.cc:1034] running on GCE node
I20260810 00:45:56.503298 7423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:56.504550 7423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:56.505805 7423 hybrid_clock.cc:648] HybridClock initialized: now 1786322756505744 us; error 33 us; skew 500 ppm
I20260810 00:45:56.506126 7423 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:45:56 dist-test-slave-1nl6 krb5kdc[7401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322756, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.6.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:56.509198 7423 init.cc:377] Logged in from keytab as oryx/127.28.6.62@KRBTEST.COM (short username oryx)
I20260810 00:45:56.510629 7423 webserver.cc:460] Webserver started at http://127.28.6.62:33555/ using document root <none> and password file <none>
I20260810 00:45:56.511377 7423 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:56.511534 7423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:56.511878 7423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:56.514784 7423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "89582f0c9b9943f2ba432c32687a6435"
format_stamp: "Formatted at 2026-08-10 00:45:56 on dist-test-slave-1nl6"
server_key: "ded4e0eb20a565428972b8b94638feac"
server_key_iv: "41a85baaee4a19dda5147d45b42d0ed1"
server_key_version: "encryptionkey@0"
I20260810 00:45:56.515739 7423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "89582f0c9b9943f2ba432c32687a6435"
format_stamp: "Formatted at 2026-08-10 00:45:56 on dist-test-slave-1nl6"
server_key: "ded4e0eb20a565428972b8b94638feac"
server_key_iv: "41a85baaee4a19dda5147d45b42d0ed1"
server_key_version: "encryptionkey@0"
I20260810 00:45:56.520787 7423 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 00:45:56.524073 7440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:56.525256 7423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 00:45:56.525436 7423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "89582f0c9b9943f2ba432c32687a6435"
format_stamp: "Formatted at 2026-08-10 00:45:56 on dist-test-slave-1nl6"
server_key: "ded4e0eb20a565428972b8b94638feac"
server_key_iv: "41a85baaee4a19dda5147d45b42d0ed1"
server_key_version: "encryptionkey@0"
I20260810 00:45:56.525657 7423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:56.541031 7423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:56.545295 7423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:56.545576 7423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:56.567601 7423 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:45123
I20260810 00:45:56.567621 7527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:45123 every 8 connection(s)
I20260810 00:45:56.568852 7423 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:45:56.572544 7528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:56.578261 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 7423
I20260810 00:45:56.578470 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:45:56.579149 28696 external_mini_cluster.cc:1428] Setting key f4fecac10a8f4f68a35892936c12d486
I20260810 00:45:56.583138 7528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89582f0c9b9943f2ba432c32687a6435: Bootstrap starting.
I20260810 00:45:56.586755 7528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89582f0c9b9943f2ba432c32687a6435: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:56.587900 7528 log.cc:824] T 00000000000000000000000000000000 P 89582f0c9b9943f2ba432c32687a6435: Log is configured to *not* fsync() on all Append() calls
I20260810 00:45:56.590837 7528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89582f0c9b9943f2ba432c32687a6435: No bootstrap required, opened a new log
Aug 10 00:45:56 dist-test-slave-1nl6 krb5kdc[7401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM
I20260810 00:45:56.601477 7528 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89582f0c9b9943f2ba432c32687a6435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89582f0c9b9943f2ba432c32687a6435" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 45123 } }
I20260810 00:45:56.601902 7528 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89582f0c9b9943f2ba432c32687a6435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:56.602010 7528 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89582f0c9b9943f2ba432c32687a6435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89582f0c9b9943f2ba432c32687a6435, State: Initialized, Role: FOLLOWER
I20260810 00:45:56.602679 7528 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89582f0c9b9943f2ba432c32687a6435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89582f0c9b9943f2ba432c32687a6435" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 45123 } }
I20260810 00:45:56.602968 7531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:56.581091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46026 (local address 127.28.6.62:45123)
0810 00:45:56.581877 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:56.581888 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:56.581927 (+ 39us) server_negotiation.cc:407] Connection header received
0810 00:45:56.583716 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:56.583738 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:56.584087 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:56.584394 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:56.586279 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:56.587638 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:56.589204 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:56.589857 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:56.589980 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:56.594561 (+ 4581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:56.594578 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:56.594596 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:56.594643 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:56.598632 (+ 3989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:56.599310 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:56.599319 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:56.599329 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:56.599557 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:56.599917 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:56.599922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:56.599926 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:56.600616 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:56.600887 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:56.601198 (+ 311us) server_negotiation.cc:299] Negotiation successful
0810 00:45:56.601606 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":396,"threads_started":1}
I20260810 00:45:56.603029 7528 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89582f0c9b9943f2ba432c32687a6435 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:45:56.603155 7528 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89582f0c9b9943f2ba432c32687a6435 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:45:56.603366 7528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89582f0c9b9943f2ba432c32687a6435 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:56.606083 7528 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89582f0c9b9943f2ba432c32687a6435 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89582f0c9b9943f2ba432c32687a6435" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 45123 } }
I20260810 00:45:56.606501 7528 leader_election.cc:302] T 00000000000000000000000000000000 P 89582f0c9b9943f2ba432c32687a6435 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89582f0c9b9943f2ba432c32687a6435; no voters:
I20260810 00:45:56.607059 7528 leader_election.cc:288] T 00000000000000000000000000000000 P 89582f0c9b9943f2ba432c32687a6435 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:45:56.607195 7535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89582f0c9b9943f2ba432c32687a6435 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:45:56.607928 7535 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89582f0c9b9943f2ba432c32687a6435 [term 1 LEADER]: Becoming Leader. State: Replica: 89582f0c9b9943f2ba432c32687a6435, State: Running, Role: LEADER
I20260810 00:45:56.608456 7535 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89582f0c9b9943f2ba432c32687a6435 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89582f0c9b9943f2ba432c32687a6435" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 45123 } }
I20260810 00:45:56.609055 7528 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89582f0c9b9943f2ba432c32687a6435 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:45:56.611184 7538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89582f0c9b9943f2ba432c32687a6435 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89582f0c9b9943f2ba432c32687a6435. Latest consensus state: current_term: 1 leader_uuid: "89582f0c9b9943f2ba432c32687a6435" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89582f0c9b9943f2ba432c32687a6435" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 45123 } } }
I20260810 00:45:56.612092 7538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89582f0c9b9943f2ba432c32687a6435 [sys.catalog]: This master's current role is: LEADER
I20260810 00:45:56.611287 7536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89582f0c9b9943f2ba432c32687a6435 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89582f0c9b9943f2ba432c32687a6435" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89582f0c9b9943f2ba432c32687a6435" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 45123 } } }
I20260810 00:45:56.612241 7536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89582f0c9b9943f2ba432c32687a6435 [sys.catalog]: This master's current role is: LEADER
W20260810 00:45:56.621093 7558 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 89582f0c9b9943f2ba432c32687a6435: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 00:45:56.621248 7558 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 00:45:56.624038 7545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:45:56.625715 7545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:45:56.633441 7545 catalog_manager.cc:1347] Generated new cluster ID: 56ba7fa6eae24df9b5e0a309e4a388c6
I20260810 00:45:56.633543 7545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:45:56.648850 7545 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:45:56.650152 7545 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:45:56.663545 7545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89582f0c9b9943f2ba432c32687a6435: Generated new TSK 0
I20260810 00:45:56.664364 7545 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.6.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.6.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.6.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.1.keytab.
Entry for principal oryx/127.28.6.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.1.keytab.
I20260810 00:45:56.730934 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.1:0
--local_ip_for_outbound_sockets=127.28.6.1
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:45123
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.1.keytab
--principal=oryx/127.28.6.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:43023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:45:57.061621 7568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:57.061884 7568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:57.062071 7568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:57.069233 7568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:57.069339 7568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:57.069473 7568 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 00:45:57.069587 7568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.1
I20260810 00:45:57.079202 7568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:43023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.1.keytab
--principal=oryx/127.28.6.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.6.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.6.62:45123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.1
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:57.080418 7568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:57.081718 7568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:57.094650 7574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:57.094726 7578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:57.094810 7575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:57.096046 7568 server_base.cc:1034] running on GCE node
I20260810 00:45:57.096623 7568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:57.097491 7568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:57.098764 7568 hybrid_clock.cc:648] HybridClock initialized: now 1786322757098693 us; error 51 us; skew 500 ppm
I20260810 00:45:57.099190 7568 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:45:57 dist-test-slave-1nl6 krb5kdc[7401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322757, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.6.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:57.102161 7568 init.cc:377] Logged in from keytab as oryx/127.28.6.1@KRBTEST.COM (short username oryx)
I20260810 00:45:57.103549 7568 webserver.cc:460] Webserver started at http://127.28.6.1:32989/ using document root <none> and password file <none>
I20260810 00:45:57.104220 7568 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:57.104360 7568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:57.104686 7568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:57.107534 7568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-0/data/instance:
uuid: "e74119537bff4b4ab0ca81a2ecc85fc8"
format_stamp: "Formatted at 2026-08-10 00:45:57 on dist-test-slave-1nl6"
server_key: "5a371e656f2e96e8cbbeefa15dfef361"
server_key_iv: "de51c3e6db6a311154cd4039d9de51e2"
server_key_version: "encryptionkey@0"
I20260810 00:45:57.108326 7568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance:
uuid: "e74119537bff4b4ab0ca81a2ecc85fc8"
format_stamp: "Formatted at 2026-08-10 00:45:57 on dist-test-slave-1nl6"
server_key: "5a371e656f2e96e8cbbeefa15dfef361"
server_key_iv: "de51c3e6db6a311154cd4039d9de51e2"
server_key_version: "encryptionkey@0"
I20260810 00:45:57.113619 7568 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 00:45:57.117053 7590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:57.118351 7568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260810 00:45:57.118520 7568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-0/wal
uuid: "e74119537bff4b4ab0ca81a2ecc85fc8"
format_stamp: "Formatted at 2026-08-10 00:45:57 on dist-test-slave-1nl6"
server_key: "5a371e656f2e96e8cbbeefa15dfef361"
server_key_iv: "de51c3e6db6a311154cd4039d9de51e2"
server_key_version: "encryptionkey@0"
I20260810 00:45:57.118732 7568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:57.135901 7568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:57.139765 7568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:57.140038 7568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:57.142544 7568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:45:57.142648 7568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:57.142776 7568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:45:57.142849 7568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 00:45:57 dist-test-slave-1nl6 krb5kdc[7401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322757, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.6.1@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM
I20260810 00:45:57.167322 7599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:57.143271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:56443 (local address 127.28.6.62:45123)
0810 00:45:57.143812 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:57.143819 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:57.144855 (+ 1036us) server_negotiation.cc:407] Connection header received
0810 00:45:57.147106 (+ 2251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:57.147112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:57.147200 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:57.147341 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:57.149433 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:57.150404 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:57.153556 (+ 3152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:57.154175 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:57.154322 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:57.159290 (+ 4968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:57.159300 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:57.159308 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:57.159360 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:57.163098 (+ 3738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:57.164130 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:57.164136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:57.164140 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:57.164268 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:57.164872 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:57.164878 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:57.164882 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:57.165314 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:57.165485 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:57.166399 (+ 914us) server_negotiation.cc:299] Negotiation successful
0810 00:45:57.166738 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":310,"threads_started":1}
I20260810 00:45:57.167822 7600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:57.143910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:45123 (local address 127.28.6.1:56443)
0810 00:45:57.144694 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:57.144729 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:57.146778 (+ 2049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:57.147580 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:57.147596 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:57.148392 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:57.149257 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:57.149277 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:57.150589 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:57.150595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:57.153412 (+ 2817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:57.153427 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:57.154444 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:57.154452 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:57.154691 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:57.155628 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:57.155665 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:57.159017 (+ 3352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:57.163287 (+ 4270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:57.163298 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:57.163316 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:57.163999 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:57.164395 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:57.164400 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:57.164404 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:57.164705 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:57.165701 (+ 996us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:57.165707 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:57.166002 (+ 295us) client_negotiation.cc:769] Sending connection context
0810 00:45:57.166227 (+ 225us) client_negotiation.cc:240] Negotiation successful
0810 00:45:57.166596 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":360,"threads_started":1}
I20260810 00:45:57.193204 7599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:57.178803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:54609 (local address 127.28.6.62:45123)
0810 00:45:57.178992 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:57.178997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:57.179355 (+ 358us) server_negotiation.cc:407] Connection header received
0810 00:45:57.179539 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:57.179543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:57.179599 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:57.179676 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:57.181047 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:57.181668 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:57.184290 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:57.184937 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:57.185087 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:57.186824 (+ 1737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:57.186833 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:57.186841 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:57.186867 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:57.189833 (+ 2966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:57.190748 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:57.190752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:57.190754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:57.190960 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:57.191508 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:57.191517 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:57.191523 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:57.191940 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:57.192120 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:57.192656 (+ 536us) server_negotiation.cc:299] Negotiation successful
0810 00:45:57.192868 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 00:45:57.193341 7713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:57.178287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:45123 (local address 127.28.6.1:54609)
0810 00:45:57.179232 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:57.179257 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:57.179422 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:57.179743 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:57.179751 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:57.180435 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:57.180903 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:57.180918 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:57.181811 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:57.181826 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:57.184003 (+ 2177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:57.184027 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:57.185156 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:57.185163 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:57.185363 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:57.185953 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:57.185970 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:57.186634 (+ 664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:57.189997 (+ 3363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:57.190001 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:57.190003 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:57.190619 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:57.191098 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:57.191105 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:57.191112 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:57.191319 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:57.192115 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:57.192125 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:57.192380 (+ 255us) client_negotiation.cc:769] Sending connection context
0810 00:45:57.192544 (+ 164us) client_negotiation.cc:240] Negotiation successful
0810 00:45:57.192865 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"thread_start_us":582,"threads_started":1}
I20260810 00:45:57.201545 7568 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.1:44791
I20260810 00:45:57.201574 7756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.1:44791 every 8 connection(s)
I20260810 00:45:57.202672 7568 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
I20260810 00:45:57.204707 7757 heartbeater.cc:347] Connected to a master server at 127.28.6.62:45123
I20260810 00:45:57.204952 7757 heartbeater.cc:464] Registering TS with master...
I20260810 00:45:57.205605 7757 heartbeater.cc:511] Master 127.28.6.62:45123 requested a full tablet report, sending...
I20260810 00:45:57.205938 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 7568
I20260810 00:45:57.206151 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance
I20260810 00:45:57.206944 28696 external_mini_cluster.cc:1428] Setting key 701d344f4504bcc2e194c58b77d4d94b
I20260810 00:45:57.207908 7465 ts_manager.cc:194] Registered new tserver with Master: e74119537bff4b4ab0ca81a2ecc85fc8 (127.28.6.1:44791)
I20260810 00:45:57.209780 7465 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.6.1@KRBTEST.COM'} at 127.28.6.1:56443
WARNING: no policy specified for oryx/127.28.6.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.6.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.6.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.2.keytab.
Entry for principal oryx/127.28.6.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.2.keytab.
I20260810 00:45:57.258121 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.2:0
--local_ip_for_outbound_sockets=127.28.6.2
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:45123
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.2.keytab
--principal=oryx/127.28.6.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:43023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:45:57.597700 7762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:57.598003 7762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:57.598165 7762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:57.605118 7762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:57.605212 7762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:57.605406 7762 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 00:45:57.605547 7762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.2
I20260810 00:45:57.615634 7762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:43023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.2.keytab
--principal=oryx/127.28.6.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.6.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.6.62:45123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.2
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:57.616823 7762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:57.618202 7762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:57.631518 7769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:57.631582 7774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:57.631683 7771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:57.632673 7762 server_base.cc:1034] running on GCE node
I20260810 00:45:57.633333 7762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:57.634214 7762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:57.635459 7762 hybrid_clock.cc:648] HybridClock initialized: now 1786322757635401 us; error 54 us; skew 500 ppm
I20260810 00:45:57.635744 7762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:45:57 dist-test-slave-1nl6 krb5kdc[7401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322757, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.6.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:57.638820 7762 init.cc:377] Logged in from keytab as oryx/127.28.6.2@KRBTEST.COM (short username oryx)
I20260810 00:45:57.640202 7762 webserver.cc:460] Webserver started at http://127.28.6.2:44819/ using document root <none> and password file <none>
I20260810 00:45:57.640864 7762 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:57.640988 7762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:57.641342 7762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:57.644250 7762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-1/data/instance:
uuid: "49ede907cb434995b3b28267e5a43baa"
format_stamp: "Formatted at 2026-08-10 00:45:57 on dist-test-slave-1nl6"
server_key: "65763d09a0479e2b3a38e07888949bbe"
server_key_iv: "2152a25a049e18f9960e457d41a6518c"
server_key_version: "encryptionkey@0"
I20260810 00:45:57.645077 7762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance:
uuid: "49ede907cb434995b3b28267e5a43baa"
format_stamp: "Formatted at 2026-08-10 00:45:57 on dist-test-slave-1nl6"
server_key: "65763d09a0479e2b3a38e07888949bbe"
server_key_iv: "2152a25a049e18f9960e457d41a6518c"
server_key_version: "encryptionkey@0"
I20260810 00:45:57.649893 7762 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 00:45:57.653167 7783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:57.654430 7762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 00:45:57.654608 7762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-1/wal
uuid: "49ede907cb434995b3b28267e5a43baa"
format_stamp: "Formatted at 2026-08-10 00:45:57 on dist-test-slave-1nl6"
server_key: "65763d09a0479e2b3a38e07888949bbe"
server_key_iv: "2152a25a049e18f9960e457d41a6518c"
server_key_version: "encryptionkey@0"
I20260810 00:45:57.654829 7762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:57.680444 7762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:57.684396 7762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:57.684674 7762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:57.687381 7762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:45:57.687510 7762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:57.687656 7762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:45:57.687745 7762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 00:45:57 dist-test-slave-1nl6 krb5kdc[7401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322757, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.6.2@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM
I20260810 00:45:57.713739 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:57.688987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:45123 (local address 127.28.6.2:43107)
0810 00:45:57.689895 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:57.689934 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:57.692022 (+ 2088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:57.692840 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:57.692850 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:57.693668 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:57.694585 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:57.694617 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:57.696373 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:57.696381 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:57.698812 (+ 2431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:57.698826 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:57.699768 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:57.699777 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:57.700069 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:57.700938 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:57.700974 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:57.704280 (+ 3306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:57.708365 (+ 4085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:57.708374 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:57.708398 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:57.709102 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:57.709563 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:57.709569 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:57.709573 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:57.709923 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:57.710711 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:57.710717 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:57.711215 (+ 498us) client_negotiation.cc:769] Sending connection context
0810 00:45:57.711551 (+ 336us) client_negotiation.cc:240] Negotiation successful
0810 00:45:57.712140 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":419,"threads_started":1}
I20260810 00:45:57.714954 7599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:57.688199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:43107 (local address 127.28.6.62:45123)
0810 00:45:57.688413 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:57.688421 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:57.690077 (+ 1656us) server_negotiation.cc:407] Connection header received
0810 00:45:57.692328 (+ 2251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:57.692338 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:57.692456 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:57.692576 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:57.694810 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:57.696163 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:57.698991 (+ 2828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:57.699509 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:57.699650 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:57.704521 (+ 4871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:57.704526 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:57.704529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:57.704563 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:57.708208 (+ 3645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:57.709260 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:57.709275 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:57.709282 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:57.709432 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:57.710054 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:57.710059 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:57.710064 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:57.710475 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:57.710633 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:57.714288 (+ 3655us) server_negotiation.cc:299] Negotiation successful
0810 00:45:57.714553 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 00:45:57.737619 7599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:57.722746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:55423 (local address 127.28.6.62:45123)
0810 00:45:57.722969 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:57.722977 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:57.723734 (+ 757us) server_negotiation.cc:407] Connection header received
0810 00:45:57.724045 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:57.724050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:57.724123 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:57.724234 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:57.725166 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:57.725887 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:57.727342 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:57.727941 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:57.728092 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:57.730714 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:57.730719 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:57.730723 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:57.730751 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:57.734195 (+ 3444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:57.735186 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:57.735193 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:57.735199 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:57.735369 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:57.735836 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:57.735844 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:57.735849 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:57.736378 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:57.736554 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:57.737043 (+ 489us) server_negotiation.cc:299] Negotiation successful
0810 00:45:57.737283 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 00:45:57.737884 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:57.722718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:45123 (local address 127.28.6.2:55423)
0810 00:45:57.723618 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:57.723643 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:57.723859 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:57.724275 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:57.724280 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:57.724732 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:57.725019 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:57.725028 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:57.726031 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:57.726036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:57.727151 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:57.727167 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:57.728164 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:57.728172 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:57.728376 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:57.729093 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:57.729118 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:57.730523 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:57.734339 (+ 3816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:57.734348 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:57.734352 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:57.734972 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:57.735490 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:57.735495 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:57.735499 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:57.735661 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:57.736545 (+ 884us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:57.736553 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:57.736772 (+ 219us) client_negotiation.cc:769] Sending connection context
0810 00:45:57.736934 (+ 162us) client_negotiation.cc:240] Negotiation successful
0810 00:45:57.737273 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":512,"threads_started":1}
I20260810 00:45:57.743820 7762 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.2:35007
I20260810 00:45:57.743870 7948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.2:35007 every 8 connection(s)
I20260810 00:45:57.745030 7762 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
I20260810 00:45:57.747143 7949 heartbeater.cc:347] Connected to a master server at 127.28.6.62:45123
I20260810 00:45:57.747376 7949 heartbeater.cc:464] Registering TS with master...
I20260810 00:45:57.748008 7949 heartbeater.cc:511] Master 127.28.6.62:45123 requested a full tablet report, sending...
I20260810 00:45:57.749251 7465 ts_manager.cc:194] Registered new tserver with Master: 49ede907cb434995b3b28267e5a43baa (127.28.6.2:35007)
I20260810 00:45:57.750018 7465 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.6.2@KRBTEST.COM'} at 127.28.6.2:43107
I20260810 00:45:57.750545 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 7762
I20260810 00:45:57.750747 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance
I20260810 00:45:57.751377 28696 external_mini_cluster.cc:1428] Setting key 4f5c17238a6db4011012ca52a2beb194
WARNING: no policy specified for oryx/127.28.6.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.6.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.6.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.3.keytab.
Entry for principal oryx/127.28.6.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.3.keytab.
I20260810 00:45:57.802148 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.3:0
--local_ip_for_outbound_sockets=127.28.6.3
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:45123
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.3.keytab
--principal=oryx/127.28.6.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:43023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:45:58.133759 7956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:45:58.134063 7956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:45:58.134219 7956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:45:58.141460 7956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:45:58.141603 7956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:45:58.141763 7956 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 00:45:58.141911 7956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.3
I20260810 00:45:58.153136 7956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:43023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.3.keytab
--principal=oryx/127.28.6.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.6.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.6.62:45123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.3
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:45:58.154539 7956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:45:58.156016 7956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:45:58.169236 7966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:58.169193 7969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:45:58.169292 7967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:45:58.172257 7956 server_base.cc:1034] running on GCE node
I20260810 00:45:58.172849 7956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:45:58.173703 7956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:45:58.174998 7956 hybrid_clock.cc:648] HybridClock initialized: now 1786322758174934 us; error 61 us; skew 500 ppm
I20260810 00:45:58.175288 7956 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:45:58 dist-test-slave-1nl6 krb5kdc[7401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322758, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.6.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:45:58.182065 7956 init.cc:377] Logged in from keytab as oryx/127.28.6.3@KRBTEST.COM (short username oryx)
I20260810 00:45:58.183629 7956 webserver.cc:460] Webserver started at http://127.28.6.3:41429/ using document root <none> and password file <none>
I20260810 00:45:58.184335 7956 fs_manager.cc:359] Metadata directory not provided
I20260810 00:45:58.184469 7956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:45:58.184806 7956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:45:58.187747 7956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-2/data/instance:
uuid: "5e4f9e7de3134a81a712da7e20a37f20"
format_stamp: "Formatted at 2026-08-10 00:45:58 on dist-test-slave-1nl6"
server_key: "6d8bee788c9853b643bac9d4705ead84"
server_key_iv: "35a5cc4bbcfd4406875ecd97788e2f1c"
server_key_version: "encryptionkey@0"
I20260810 00:45:58.188560 7956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance:
uuid: "5e4f9e7de3134a81a712da7e20a37f20"
format_stamp: "Formatted at 2026-08-10 00:45:58 on dist-test-slave-1nl6"
server_key: "6d8bee788c9853b643bac9d4705ead84"
server_key_iv: "35a5cc4bbcfd4406875ecd97788e2f1c"
server_key_version: "encryptionkey@0"
I20260810 00:45:58.193606 7956 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 00:45:58.196910 7978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:58.198062 7956 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 00:45:58.198243 7956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-2/wal
uuid: "5e4f9e7de3134a81a712da7e20a37f20"
format_stamp: "Formatted at 2026-08-10 00:45:58 on dist-test-slave-1nl6"
server_key: "6d8bee788c9853b643bac9d4705ead84"
server_key_iv: "35a5cc4bbcfd4406875ecd97788e2f1c"
server_key_version: "encryptionkey@0"
I20260810 00:45:58.198458 7956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:45:58.212613 7757 heartbeater.cc:503] Master 127.28.6.62:45123 was elected leader, sending a full tablet report...
I20260810 00:45:58.216624 7956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:45:58.220316 7956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:45:58.220530 7956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:45:58.223362 7956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:45:58.223488 7956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:45:58.223623 7956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:45:58.223695 7956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 00:45:58 dist-test-slave-1nl6 krb5kdc[7401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322758, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.6.3@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM
I20260810 00:45:58.246444 7599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:58.223713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:53993 (local address 127.28.6.62:45123)
0810 00:45:58.223871 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:58.223876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:58.225233 (+ 1357us) server_negotiation.cc:407] Connection header received
0810 00:45:58.227631 (+ 2398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:58.227636 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:58.227735 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:58.227855 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:58.230079 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:58.231149 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:58.233066 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:58.233488 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:58.233622 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:58.238085 (+ 4463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:58.238107 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:58.238114 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:58.238440 (+ 326us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:58.242097 (+ 3657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:58.242957 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:58.242964 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:58.242968 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:58.243114 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:58.243827 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:58.243836 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:58.243842 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:58.244470 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:58.244651 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:58.245633 (+ 982us) server_negotiation.cc:299] Negotiation successful
0810 00:45:58.245905 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 00:45:58.247220 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:58.224280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:45123 (local address 127.28.6.3:53993)
0810 00:45:58.225088 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:58.225126 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:58.227316 (+ 2190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:58.228187 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:58.228202 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:58.229004 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:58.229902 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:58.229923 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:58.231353 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:58.231364 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:58.232917 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:58.232930 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:58.233734 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:58.233743 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:58.233962 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:58.234626 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:58.234651 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:58.237787 (+ 3136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:58.242245 (+ 4458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:58.242253 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:58.242267 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:58.242813 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:58.243243 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:58.243250 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:58.243254 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:58.243638 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:58.244628 (+ 990us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:58.244637 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:58.245023 (+ 386us) client_negotiation.cc:769] Sending connection context
0810 00:45:58.245426 (+ 403us) client_negotiation.cc:240] Negotiation successful
0810 00:45:58.245965 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":353,"threads_started":1}
I20260810 00:45:58.269649 8069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:58.253633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:45123 (local address 127.28.6.3:56145)
0810 00:45:58.254392 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:58.254418 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:58.254578 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:58.254986 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:58.254995 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:58.255703 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:58.256156 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:58.256171 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:58.257172 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:58.257180 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:58.259223 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:58.259249 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:58.260200 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:58.260209 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:58.260512 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:58.261179 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:58.261210 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:58.262315 (+ 1105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:58.266889 (+ 4574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:58.266939 (+ 50us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:58.266948 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:58.267458 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:58.267935 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:58.267939 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:58.267941 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:58.268055 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:58.268761 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:58.268764 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:58.268919 (+ 155us) client_negotiation.cc:769] Sending connection context
0810 00:45:58.269018 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 00:45:58.269230 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":416,"threads_started":1}
I20260810 00:45:58.269889 7599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:58.253914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:56145 (local address 127.28.6.62:45123)
0810 00:45:58.254131 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:58.254138 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:58.254526 (+ 388us) server_negotiation.cc:407] Connection header received
0810 00:45:58.254710 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:58.254716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:58.254803 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:58.254889 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:58.256365 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:58.257037 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:58.259457 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:58.259995 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:58.260142 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:58.262562 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:58.262569 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:58.262574 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:58.262616 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:58.266719 (+ 4103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:58.267618 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:58.267626 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:58.267632 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:58.267803 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:58.268176 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:58.268183 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:58.268189 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:58.268639 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:58.268820 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:58.269212 (+ 392us) server_negotiation.cc:299] Negotiation successful
0810 00:45:58.269505 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 00:45:58.280351 7956 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.3:41251
I20260810 00:45:58.280388 8151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.3:41251 every 8 connection(s)
I20260810 00:45:58.281533 7956 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
I20260810 00:45:58.282858 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 7956
I20260810 00:45:58.283078 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance
I20260810 00:45:58.283650 28696 external_mini_cluster.cc:1428] Setting key 47a1c452a6b2799c6990e3fe5a7487ae
I20260810 00:45:58.283663 8152 heartbeater.cc:347] Connected to a master server at 127.28.6.62:45123
I20260810 00:45:58.283892 8152 heartbeater.cc:464] Registering TS with master...
I20260810 00:45:58.284590 8152 heartbeater.cc:511] Master 127.28.6.62:45123 requested a full tablet report, sending...
I20260810 00:45:58.286103 7463 ts_manager.cc:194] Registered new tserver with Master: 5e4f9e7de3134a81a712da7e20a37f20 (127.28.6.3:41251)
I20260810 00:45:58.287212 7463 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.6.3@KRBTEST.COM'} at 127.28.6.3:53993
I20260810 00:45:58.298614 28696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 00:45:58 dist-test-slave-1nl6 krb5kdc[7401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322758, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 00:45:58 dist-test-slave-1nl6 krb5kdc[7401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:45:58 dist-test-slave-1nl6 krb5kdc[7401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 00:45:58.369488 7599 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:45:58.357543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58124 (local address 127.28.6.62:45123)
0810 00:45:58.357879 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:58.357889 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:58.358063 (+ 174us) server_negotiation.cc:407] Connection header received
0810 00:45:58.358287 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:58.358294 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:58.358435 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:58.358548 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:58.360103 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:58.360832 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:58.362089 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:58.362513 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:58.362666 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:58.369171 (+ 6505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58124: BlockingRecv error: recv got EOF from 127.0.0.1:58124 (error 108)
Metrics: {"server-negotiator.queue_time_us":186}
Aug 10 00:45:58 dist-test-slave-1nl6 krb5kdc[7401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322758, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM
I20260810 00:45:58.385766 7599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:58.372905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58134 (local address 127.28.6.62:45123)
0810 00:45:58.373111 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:58.373119 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:58.373411 (+ 292us) server_negotiation.cc:407] Connection header received
0810 00:45:58.373585 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:58.373591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:58.373673 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:58.373775 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:58.374895 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:58.375671 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:58.376964 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:58.377402 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:58.377556 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:58.381198 (+ 3642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:58.381203 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:58.381206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:58.381237 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:58.383666 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:58.384257 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:58.384262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:58.384264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:58.384389 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:58.384710 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:58.384714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:58.384716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:58.384966 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:58.385181 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:58.385391 (+ 210us) server_negotiation.cc:299] Negotiation successful
0810 00:45:58.385543 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 00:45:58.389652 7463 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58134:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 00:45:58.396471 7463 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 00:45:58.421797 8177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:58.408738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53188 (local address 127.28.6.2:35007)
0810 00:45:58.413333 (+ 4595us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:58.413343 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:58.413999 (+ 656us) server_negotiation.cc:407] Connection header received
0810 00:45:58.414110 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:58.414118 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:58.414456 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:58.414751 (+ 295us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:58.415727 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:58.417296 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:58.419799 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:58.420654 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:58.420803 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:58.421024 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:58.421345 (+ 321us) server_negotiation.cc:299] Negotiation successful
0810 00:45:58.421515 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4429,"thread_start_us":4294,"threads_started":1}
I20260810 00:45:58.421977 8174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:58.408190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:35007 (local address 127.0.0.1:53188)
0810 00:45:58.413558 (+ 5368us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:58.413618 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:58.413812 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:58.415007 (+ 1195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:58.415014 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:58.415046 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:58.415479 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:58.415490 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:58.417454 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:58.417459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:58.419639 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:58.419652 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:58.420892 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:58.420901 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:58.421115 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:58.421151 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 00:45:58.421277 (+ 126us) client_negotiation.cc:240] Negotiation successful
0810 00:45:58.421460 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5026,"mutex_wait_us":35,"thread_start_us":5349,"threads_started":1}
I20260810 00:45:58.423532 8173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:58.408387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:41251 (local address 127.0.0.1:60896)
0810 00:45:58.413824 (+ 5437us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:58.413842 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:58.413961 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:58.415465 (+ 1504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:58.415469 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:58.415493 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:58.415906 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:58.415919 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:58.418158 (+ 2239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:58.418166 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:58.420747 (+ 2581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:58.420763 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:58.422692 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:58.422700 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:58.422874 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:58.423032 (+ 158us) client_negotiation.cc:769] Sending connection context
0810 00:45:58.423134 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 00:45:58.423262 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5313,"mutex_wait_us":104,"thread_start_us":4673,"threads_started":1}
I20260810 00:45:58.424489 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:58.408675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60896 (local address 127.28.6.3:41251)
0810 00:45:58.409456 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:58.409484 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:58.414854 (+ 5370us) server_negotiation.cc:407] Connection header received
0810 00:45:58.415008 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:58.415015 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:58.415231 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:58.416110 (+ 879us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:58.416417 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:58.418005 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:58.420944 (+ 2939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:58.422431 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:58.422561 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:58.422785 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:58.423966 (+ 1181us) server_negotiation.cc:299] Negotiation successful
0810 00:45:58.424137 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":442,"threads_started":1}
I20260810 00:45:58.425241 8178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:58.414006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:44791 (local address 127.0.0.1:47078)
0810 00:45:58.414564 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:58.414583 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:58.414693 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:58.416272 (+ 1579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:58.416277 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:58.416301 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:58.416684 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:58.416693 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:58.418676 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:58.418682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:58.422247 (+ 3565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:58.422262 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:58.424198 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:58.424208 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:58.424480 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:58.424554 (+ 74us) client_negotiation.cc:769] Sending connection context
0810 00:45:58.424684 (+ 130us) client_negotiation.cc:240] Negotiation successful
0810 00:45:58.424873 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":1206,"thread_start_us":294,"threads_started":1}
I20260810 00:45:58.425278 8179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:58.414326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47078 (local address 127.28.6.1:44791)
0810 00:45:58.415848 (+ 1522us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:58.415855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:58.415876 (+ 21us) server_negotiation.cc:407] Connection header received
0810 00:45:58.415954 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:58.415958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:58.416091 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:58.416291 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:58.416855 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:58.418527 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:58.422435 (+ 3908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:58.423949 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:58.424108 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:58.424354 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:58.424818 (+ 464us) server_negotiation.cc:299] Negotiation successful
0810 00:45:58.425007 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1385,"thread_start_us":1244,"threads_started":1}
I20260810 00:45:58.428483 7657 tablet_service.cc:1467] Processing CreateTablet for tablet 4f10f8b2d3ac43db80d8b03ab1d9eaaa (DEFAULT_TABLE table=test-table [id=782400e1c9734bd0bb3686172334aa4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:58.428499 8049 tablet_service.cc:1467] Processing CreateTablet for tablet 4f10f8b2d3ac43db80d8b03ab1d9eaaa (DEFAULT_TABLE table=test-table [id=782400e1c9734bd0bb3686172334aa4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:58.428871 7853 tablet_service.cc:1467] Processing CreateTablet for tablet 4f10f8b2d3ac43db80d8b03ab1d9eaaa (DEFAULT_TABLE table=test-table [id=782400e1c9734bd0bb3686172334aa4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:58.429895 7657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f10f8b2d3ac43db80d8b03ab1d9eaaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:58.429948 8049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f10f8b2d3ac43db80d8b03ab1d9eaaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:58.430116 7853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f10f8b2d3ac43db80d8b03ab1d9eaaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:58.441139 8180 tablet_bootstrap.cc:492] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P e74119537bff4b4ab0ca81a2ecc85fc8: Bootstrap starting.
I20260810 00:45:58.445048 8180 tablet_bootstrap.cc:654] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P e74119537bff4b4ab0ca81a2ecc85fc8: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:58.446099 8180 log.cc:824] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P e74119537bff4b4ab0ca81a2ecc85fc8: Log is configured to *not* fsync() on all Append() calls
I20260810 00:45:58.449085 8180 tablet_bootstrap.cc:492] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P e74119537bff4b4ab0ca81a2ecc85fc8: No bootstrap required, opened a new log
I20260810 00:45:58.449374 8180 ts_tablet_manager.cc:1397] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P e74119537bff4b4ab0ca81a2ecc85fc8: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260810 00:45:58.449942 8182 tablet_bootstrap.cc:492] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 5e4f9e7de3134a81a712da7e20a37f20: Bootstrap starting.
I20260810 00:45:58.450201 8181 tablet_bootstrap.cc:492] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 49ede907cb434995b3b28267e5a43baa: Bootstrap starting.
I20260810 00:45:58.453593 8181 tablet_bootstrap.cc:654] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 49ede907cb434995b3b28267e5a43baa: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:58.453595 8182 tablet_bootstrap.cc:654] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 5e4f9e7de3134a81a712da7e20a37f20: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:58.454838 8181 log.cc:824] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 49ede907cb434995b3b28267e5a43baa: Log is configured to *not* fsync() on all Append() calls
I20260810 00:45:58.454873 8182 log.cc:824] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 5e4f9e7de3134a81a712da7e20a37f20: Log is configured to *not* fsync() on all Append() calls
I20260810 00:45:58.457414 8181 tablet_bootstrap.cc:492] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 49ede907cb434995b3b28267e5a43baa: No bootstrap required, opened a new log
I20260810 00:45:58.457690 8181 ts_tablet_manager.cc:1397] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 49ede907cb434995b3b28267e5a43baa: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260810 00:45:58.457885 8182 tablet_bootstrap.cc:492] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 5e4f9e7de3134a81a712da7e20a37f20: No bootstrap required, opened a new log
I20260810 00:45:58.458189 8182 ts_tablet_manager.cc:1397] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 5e4f9e7de3134a81a712da7e20a37f20: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 00:45:58.458500 8180 raft_consensus.cc:348] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P e74119537bff4b4ab0ca81a2ecc85fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } } peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } }
I20260810 00:45:58.459002 8180 raft_consensus.cc:374] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P e74119537bff4b4ab0ca81a2ecc85fc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:58.459124 8180 raft_consensus.cc:729] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P e74119537bff4b4ab0ca81a2ecc85fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e74119537bff4b4ab0ca81a2ecc85fc8, State: Initialized, Role: FOLLOWER
I20260810 00:45:58.459834 8180 consensus_queue.cc:260] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P e74119537bff4b4ab0ca81a2ecc85fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } } peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } }
I20260810 00:45:58.461540 8180 ts_tablet_manager.cc:1428] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P e74119537bff4b4ab0ca81a2ecc85fc8: Time spent starting tablet: real 0.012s user 0.002s sys 0.009s
I20260810 00:45:58.466560 8181 raft_consensus.cc:348] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 49ede907cb434995b3b28267e5a43baa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } } peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } }
I20260810 00:45:58.466934 8181 raft_consensus.cc:374] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 49ede907cb434995b3b28267e5a43baa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:58.467046 8181 raft_consensus.cc:729] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 49ede907cb434995b3b28267e5a43baa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49ede907cb434995b3b28267e5a43baa, State: Initialized, Role: FOLLOWER
I20260810 00:45:58.467614 8181 consensus_queue.cc:260] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 49ede907cb434995b3b28267e5a43baa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } } peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } }
I20260810 00:45:58.468178 8182 raft_consensus.cc:348] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 5e4f9e7de3134a81a712da7e20a37f20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } } peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } }
I20260810 00:45:58.468516 8182 raft_consensus.cc:374] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 5e4f9e7de3134a81a712da7e20a37f20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:58.468631 8182 raft_consensus.cc:729] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 5e4f9e7de3134a81a712da7e20a37f20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e4f9e7de3134a81a712da7e20a37f20, State: Initialized, Role: FOLLOWER
I20260810 00:45:58.468621 7949 heartbeater.cc:503] Master 127.28.6.62:45123 was elected leader, sending a full tablet report...
I20260810 00:45:58.469349 8181 ts_tablet_manager.cc:1428] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 49ede907cb434995b3b28267e5a43baa: Time spent starting tablet: real 0.012s user 0.004s sys 0.010s
I20260810 00:45:58.469170 8182 consensus_queue.cc:260] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 5e4f9e7de3134a81a712da7e20a37f20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } } peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } }
I20260810 00:45:58.470367 8152 heartbeater.cc:503] Master 127.28.6.62:45123 was elected leader, sending a full tablet report...
I20260810 00:45:58.470993 8182 ts_tablet_manager.cc:1428] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 5e4f9e7de3134a81a712da7e20a37f20: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260810 00:45:58.485526 8187 raft_consensus.cc:482] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P e74119537bff4b4ab0ca81a2ecc85fc8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:45:58.485803 8187 raft_consensus.cc:504] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P e74119537bff4b4ab0ca81a2ecc85fc8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } } peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } }
I20260810 00:45:58.487392 8187 leader_election.cc:288] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P e74119537bff4b4ab0ca81a2ecc85fc8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49ede907cb434995b3b28267e5a43baa (127.28.6.2:35007), 5e4f9e7de3134a81a712da7e20a37f20 (127.28.6.3:41251)
I20260810 00:45:58.495769 8177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:58.487820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:50455 (local address 127.28.6.2:35007)
0810 00:45:58.488027 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:58.488034 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:58.488888 (+ 854us) server_negotiation.cc:407] Connection header received
0810 00:45:58.489092 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:58.489098 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:58.489169 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:58.489267 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:58.490104 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:58.491227 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:58.493360 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:58.494694 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:58.494856 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:58.494995 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:58.495373 (+ 378us) server_negotiation.cc:299] Negotiation successful
0810 00:45:58.495499 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 00:45:58.495795 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:58.487863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:43499 (local address 127.28.6.3:41251)
0810 00:45:58.488024 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:58.488029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:58.488955 (+ 926us) server_negotiation.cc:407] Connection header received
0810 00:45:58.489092 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:58.489098 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:58.489169 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:58.489267 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:58.490104 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:58.491227 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:58.493379 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:58.494837 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:58.495034 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:58.495141 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:58.495416 (+ 275us) server_negotiation.cc:299] Negotiation successful
0810 00:45:58.495551 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 00:45:58.496073 8193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:58.487853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:35007 (local address 127.28.6.1:50455)
0810 00:45:58.488772 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:58.488792 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:58.488949 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:58.489305 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:58.489310 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:58.489347 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:58.489978 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:58.489990 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:58.491426 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:58.491430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:58.493186 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:58.493200 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:58.494990 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:58.494997 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:58.495180 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:58.495222 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 00:45:58.495299 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 00:45:58.495437 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"mutex_wait_us":83,"spinlock_wait_cycles":896,"thread_start_us":437,"threads_started":1}
I20260810 00:45:58.496105 8194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:58.487853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:41251 (local address 127.28.6.1:43499)
0810 00:45:58.488849 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:58.488866 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:58.488959 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:58.489305 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:58.489310 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:58.489353 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:58.489878 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:58.489901 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:58.491372 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:58.491377 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:58.493207 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:58.493227 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:58.495116 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:58.495121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:58.495231 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:58.495254 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 00:45:58.495371 (+ 117us) client_negotiation.cc:240] Negotiation successful
0810 00:45:58.495497 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":871,"mutex_wait_us":305,"spinlock_wait_cycles":5120,"thread_start_us":511,"threads_started":1}
W20260810 00:45:58.496985 7953 tablet.cc:2369] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 49ede907cb434995b3b28267e5a43baa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:45:58.497157 7880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f10f8b2d3ac43db80d8b03ab1d9eaaa" candidate_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49ede907cb434995b3b28267e5a43baa" is_pre_election: true
I20260810 00:45:58.497288 8082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f10f8b2d3ac43db80d8b03ab1d9eaaa" candidate_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e4f9e7de3134a81a712da7e20a37f20" is_pre_election: true
I20260810 00:45:58.497746 7880 raft_consensus.cc:2454] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 49ede907cb434995b3b28267e5a43baa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e74119537bff4b4ab0ca81a2ecc85fc8 in term 0.
I20260810 00:45:58.497740 8082 raft_consensus.cc:2454] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 5e4f9e7de3134a81a712da7e20a37f20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e74119537bff4b4ab0ca81a2ecc85fc8 in term 0.
I20260810 00:45:58.498447 7592 leader_election.cc:302] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P e74119537bff4b4ab0ca81a2ecc85fc8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 49ede907cb434995b3b28267e5a43baa, 5e4f9e7de3134a81a712da7e20a37f20, e74119537bff4b4ab0ca81a2ecc85fc8; no voters:
I20260810 00:45:58.498827 8187 raft_consensus.cc:2781] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P e74119537bff4b4ab0ca81a2ecc85fc8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 00:45:58.499009 8187 raft_consensus.cc:482] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P e74119537bff4b4ab0ca81a2ecc85fc8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 00:45:58.499135 8187 raft_consensus.cc:3037] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P e74119537bff4b4ab0ca81a2ecc85fc8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:58.501322 8187 raft_consensus.cc:504] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P e74119537bff4b4ab0ca81a2ecc85fc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } } peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } }
I20260810 00:45:58.502007 8187 leader_election.cc:288] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P e74119537bff4b4ab0ca81a2ecc85fc8 [CANDIDATE]: Term 1 election: Requested vote from peers 49ede907cb434995b3b28267e5a43baa (127.28.6.2:35007), 5e4f9e7de3134a81a712da7e20a37f20 (127.28.6.3:41251)
I20260810 00:45:58.502362 7880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f10f8b2d3ac43db80d8b03ab1d9eaaa" candidate_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49ede907cb434995b3b28267e5a43baa"
I20260810 00:45:58.502542 8082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f10f8b2d3ac43db80d8b03ab1d9eaaa" candidate_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e4f9e7de3134a81a712da7e20a37f20"
I20260810 00:45:58.502566 7880 raft_consensus.cc:3037] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 49ede907cb434995b3b28267e5a43baa [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:58.502776 8082 raft_consensus.cc:3037] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 5e4f9e7de3134a81a712da7e20a37f20 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:58.504971 7880 raft_consensus.cc:2454] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 49ede907cb434995b3b28267e5a43baa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e74119537bff4b4ab0ca81a2ecc85fc8 in term 1.
I20260810 00:45:58.505119 8082 raft_consensus.cc:2454] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 5e4f9e7de3134a81a712da7e20a37f20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e74119537bff4b4ab0ca81a2ecc85fc8 in term 1.
I20260810 00:45:58.505393 7592 leader_election.cc:302] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P e74119537bff4b4ab0ca81a2ecc85fc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49ede907cb434995b3b28267e5a43baa, e74119537bff4b4ab0ca81a2ecc85fc8; no voters:
I20260810 00:45:58.505672 8187 raft_consensus.cc:2781] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P e74119537bff4b4ab0ca81a2ecc85fc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:45:58.506068 8187 raft_consensus.cc:686] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P e74119537bff4b4ab0ca81a2ecc85fc8 [term 1 LEADER]: Becoming Leader. State: Replica: e74119537bff4b4ab0ca81a2ecc85fc8, State: Running, Role: LEADER
I20260810 00:45:58.506546 8187 consensus_queue.cc:237] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P e74119537bff4b4ab0ca81a2ecc85fc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } } peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } }
I20260810 00:45:58.511059 7463 catalog_manager.cc:5495] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P e74119537bff4b4ab0ca81a2ecc85fc8 reported cstate change: term changed from 0 to 1, leader changed from <none> to e74119537bff4b4ab0ca81a2ecc85fc8 (127.28.6.1). New cstate: current_term: 1 leader_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } health_report { overall_health: HEALTHY } } }
W20260810 00:45:58.533111 8153 tablet.cc:2369] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 5e4f9e7de3134a81a712da7e20a37f20: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:45:58.545979 8179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:58.540984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47084 (local address 127.28.6.1:44791)
0810 00:45:58.541215 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:58.541221 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:58.541240 (+ 19us) server_negotiation.cc:407] Connection header received
0810 00:45:58.541452 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:58.541455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:58.541550 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:58.541674 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:45:58.542014 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:58.542694 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:58.544005 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:58.544406 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:58.544502 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:58.544903 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:45:58.545396 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:45:58.545501 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:58.545604 (+ 103us) server_negotiation.cc:299] Negotiation successful
0810 00:45:58.545665 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260810 00:45:58.554044 7880 raft_consensus.cc:1261] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 49ede907cb434995b3b28267e5a43baa [term 1 FOLLOWER]: Refusing update from remote peer e74119537bff4b4ab0ca81a2ecc85fc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:45:58.554152 8082 raft_consensus.cc:1261] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 5e4f9e7de3134a81a712da7e20a37f20 [term 1 FOLLOWER]: Refusing update from remote peer e74119537bff4b4ab0ca81a2ecc85fc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:45:58.555171 8195 consensus_queue.cc:1035] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P e74119537bff4b4ab0ca81a2ecc85fc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:45:58.555526 8187 consensus_queue.cc:1035] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P e74119537bff4b4ab0ca81a2ecc85fc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:45:58.592075 7463 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:45:58.592377 7463 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:45:58.596349 7463 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 89582f0c9b9943f2ba432c32687a6435: Sending DeleteTablet for 3 replicas of tablet 4f10f8b2d3ac43db80d8b03ab1d9eaaa
I20260810 00:45:58.597440 7853 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f10f8b2d3ac43db80d8b03ab1d9eaaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:45:58 UTC) from {username='oryx', principal='oryx/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:53188
I20260810 00:45:58.597936 8049 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f10f8b2d3ac43db80d8b03ab1d9eaaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:45:58 UTC) from {username='oryx', principal='oryx/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:60896
I20260810 00:45:58.598559 7657 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f10f8b2d3ac43db80d8b03ab1d9eaaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:45:58 UTC) from {username='oryx', principal='oryx/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:47078
I20260810 00:45:58.599141 8215 tablet_replica.cc:330] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 5e4f9e7de3134a81a712da7e20a37f20: stopping tablet replica
I20260810 00:45:58.599541 8215 raft_consensus.cc:2229] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 5e4f9e7de3134a81a712da7e20a37f20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:45:58.600092 8215 raft_consensus.cc:2258] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 5e4f9e7de3134a81a712da7e20a37f20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:45:58.600384 7463 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58134:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 00:45:58.600867 8214 tablet_replica.cc:330] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 49ede907cb434995b3b28267e5a43baa: stopping tablet replica
W20260810 00:45:58.601265 7463 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 00:45:58.601251 8214 raft_consensus.cc:2229] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 49ede907cb434995b3b28267e5a43baa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:45:58.601702 8214 raft_consensus.cc:2258] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 49ede907cb434995b3b28267e5a43baa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:45:58.603652 8215 ts_tablet_manager.cc:1905] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 5e4f9e7de3134a81a712da7e20a37f20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:45:58.603678 8216 tablet_replica.cc:330] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P e74119537bff4b4ab0ca81a2ecc85fc8: stopping tablet replica
I20260810 00:45:58.604050 8216 raft_consensus.cc:2229] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P e74119537bff4b4ab0ca81a2ecc85fc8 [term 1 LEADER]: Raft consensus shutting down.
I20260810 00:45:58.604712 8216 raft_consensus.cc:2258] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P e74119537bff4b4ab0ca81a2ecc85fc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:45:58.605365 8214 ts_tablet_manager.cc:1905] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 49ede907cb434995b3b28267e5a43baa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:45:58.610059 7853 tablet_service.cc:1467] Processing CreateTablet for tablet 10a780a47e63441895d181e08860d516 (DEFAULT_TABLE table=test-table [id=2705ea9efd0c414fb5d1768aebd9e212]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:58.610545 7853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10a780a47e63441895d181e08860d516. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:58.610692 8215 ts_tablet_manager.cc:1918] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 5e4f9e7de3134a81a712da7e20a37f20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:45:58.610862 8215 log.cc:1196] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 5e4f9e7de3134a81a712da7e20a37f20: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-2/wal/wals/4f10f8b2d3ac43db80d8b03ab1d9eaaa
I20260810 00:45:58.611483 8215 ts_tablet_manager.cc:1939] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 5e4f9e7de3134a81a712da7e20a37f20: Deleting consensus metadata
I20260810 00:45:58.613617 7441 catalog_manager.cc:4841] TS 5e4f9e7de3134a81a712da7e20a37f20 (127.28.6.3:41251): tablet 4f10f8b2d3ac43db80d8b03ab1d9eaaa (table test-table [id=782400e1c9734bd0bb3686172334aa4b]) successfully deleted
I20260810 00:45:58.613991 7657 tablet_service.cc:1467] Processing CreateTablet for tablet 10a780a47e63441895d181e08860d516 (DEFAULT_TABLE table=test-table [id=2705ea9efd0c414fb5d1768aebd9e212]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:58.614354 8214 ts_tablet_manager.cc:1918] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 49ede907cb434995b3b28267e5a43baa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:45:58.614471 7657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10a780a47e63441895d181e08860d516. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:58.614626 8214 log.cc:1196] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 49ede907cb434995b3b28267e5a43baa: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-1/wal/wals/4f10f8b2d3ac43db80d8b03ab1d9eaaa
I20260810 00:45:58.614624 8049 tablet_service.cc:1467] Processing CreateTablet for tablet 10a780a47e63441895d181e08860d516 (DEFAULT_TABLE table=test-table [id=2705ea9efd0c414fb5d1768aebd9e212]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:58.615131 8049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10a780a47e63441895d181e08860d516. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:58.615396 8214 ts_tablet_manager.cc:1939] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P 49ede907cb434995b3b28267e5a43baa: Deleting consensus metadata
I20260810 00:45:58.615507 8216 ts_tablet_manager.cc:1905] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P e74119537bff4b4ab0ca81a2ecc85fc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:45:58.617319 7442 catalog_manager.cc:4841] TS 49ede907cb434995b3b28267e5a43baa (127.28.6.2:35007): tablet 4f10f8b2d3ac43db80d8b03ab1d9eaaa (table test-table [id=782400e1c9734bd0bb3686172334aa4b]) successfully deleted
I20260810 00:45:58.623315 8180 tablet_bootstrap.cc:492] T 10a780a47e63441895d181e08860d516 P e74119537bff4b4ab0ca81a2ecc85fc8: Bootstrap starting.
I20260810 00:45:58.625010 8180 tablet_bootstrap.cc:654] T 10a780a47e63441895d181e08860d516 P e74119537bff4b4ab0ca81a2ecc85fc8: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:58.625370 8182 tablet_bootstrap.cc:492] T 10a780a47e63441895d181e08860d516 P 5e4f9e7de3134a81a712da7e20a37f20: Bootstrap starting.
I20260810 00:45:58.625598 8181 tablet_bootstrap.cc:492] T 10a780a47e63441895d181e08860d516 P 49ede907cb434995b3b28267e5a43baa: Bootstrap starting.
I20260810 00:45:58.627148 8182 tablet_bootstrap.cc:654] T 10a780a47e63441895d181e08860d516 P 5e4f9e7de3134a81a712da7e20a37f20: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:58.627197 8181 tablet_bootstrap.cc:654] T 10a780a47e63441895d181e08860d516 P 49ede907cb434995b3b28267e5a43baa: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:58.629140 8216 ts_tablet_manager.cc:1918] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P e74119537bff4b4ab0ca81a2ecc85fc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:45:58.629442 8216 log.cc:1196] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P e74119537bff4b4ab0ca81a2ecc85fc8: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-0/wal/wals/4f10f8b2d3ac43db80d8b03ab1d9eaaa
I20260810 00:45:58.629971 8216 ts_tablet_manager.cc:1939] T 4f10f8b2d3ac43db80d8b03ab1d9eaaa P e74119537bff4b4ab0ca81a2ecc85fc8: Deleting consensus metadata
I20260810 00:45:58.632750 7441 catalog_manager.cc:4841] TS e74119537bff4b4ab0ca81a2ecc85fc8 (127.28.6.1:44791): tablet 4f10f8b2d3ac43db80d8b03ab1d9eaaa (table test-table [id=782400e1c9734bd0bb3686172334aa4b]) successfully deleted
I20260810 00:45:58.639459 8180 tablet_bootstrap.cc:492] T 10a780a47e63441895d181e08860d516 P e74119537bff4b4ab0ca81a2ecc85fc8: No bootstrap required, opened a new log
I20260810 00:45:58.639617 8180 ts_tablet_manager.cc:1397] T 10a780a47e63441895d181e08860d516 P e74119537bff4b4ab0ca81a2ecc85fc8: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.004s
I20260810 00:45:58.640471 8180 raft_consensus.cc:348] T 10a780a47e63441895d181e08860d516 P e74119537bff4b4ab0ca81a2ecc85fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } }
I20260810 00:45:58.640713 8180 raft_consensus.cc:374] T 10a780a47e63441895d181e08860d516 P e74119537bff4b4ab0ca81a2ecc85fc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:58.640820 8180 raft_consensus.cc:729] T 10a780a47e63441895d181e08860d516 P e74119537bff4b4ab0ca81a2ecc85fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e74119537bff4b4ab0ca81a2ecc85fc8, State: Initialized, Role: FOLLOWER
I20260810 00:45:58.641108 8180 consensus_queue.cc:260] T 10a780a47e63441895d181e08860d516 P e74119537bff4b4ab0ca81a2ecc85fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } }
I20260810 00:45:58.641808 8182 tablet_bootstrap.cc:492] T 10a780a47e63441895d181e08860d516 P 5e4f9e7de3134a81a712da7e20a37f20: No bootstrap required, opened a new log
I20260810 00:45:58.642045 8182 ts_tablet_manager.cc:1397] T 10a780a47e63441895d181e08860d516 P 5e4f9e7de3134a81a712da7e20a37f20: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.010s
I20260810 00:45:58.644074 8180 ts_tablet_manager.cc:1428] T 10a780a47e63441895d181e08860d516 P e74119537bff4b4ab0ca81a2ecc85fc8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260810 00:45:58.644575 8182 raft_consensus.cc:348] T 10a780a47e63441895d181e08860d516 P 5e4f9e7de3134a81a712da7e20a37f20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } }
I20260810 00:45:58.644965 8182 raft_consensus.cc:374] T 10a780a47e63441895d181e08860d516 P 5e4f9e7de3134a81a712da7e20a37f20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:58.645134 8182 raft_consensus.cc:729] T 10a780a47e63441895d181e08860d516 P 5e4f9e7de3134a81a712da7e20a37f20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e4f9e7de3134a81a712da7e20a37f20, State: Initialized, Role: FOLLOWER
I20260810 00:45:58.645552 8182 consensus_queue.cc:260] T 10a780a47e63441895d181e08860d516 P 5e4f9e7de3134a81a712da7e20a37f20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } }
I20260810 00:45:58.646872 8182 ts_tablet_manager.cc:1428] T 10a780a47e63441895d181e08860d516 P 5e4f9e7de3134a81a712da7e20a37f20: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20260810 00:45:58.647648 8181 tablet_bootstrap.cc:492] T 10a780a47e63441895d181e08860d516 P 49ede907cb434995b3b28267e5a43baa: No bootstrap required, opened a new log
I20260810 00:45:58.647801 8181 ts_tablet_manager.cc:1397] T 10a780a47e63441895d181e08860d516 P 49ede907cb434995b3b28267e5a43baa: Time spent bootstrapping tablet: real 0.022s user 0.016s sys 0.005s
I20260810 00:45:58.648593 8181 raft_consensus.cc:348] T 10a780a47e63441895d181e08860d516 P 49ede907cb434995b3b28267e5a43baa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } }
I20260810 00:45:58.648840 8181 raft_consensus.cc:374] T 10a780a47e63441895d181e08860d516 P 49ede907cb434995b3b28267e5a43baa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:58.648977 8181 raft_consensus.cc:729] T 10a780a47e63441895d181e08860d516 P 49ede907cb434995b3b28267e5a43baa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49ede907cb434995b3b28267e5a43baa, State: Initialized, Role: FOLLOWER
I20260810 00:45:58.649267 8181 consensus_queue.cc:260] T 10a780a47e63441895d181e08860d516 P 49ede907cb434995b3b28267e5a43baa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } }
I20260810 00:45:58.650468 8181 ts_tablet_manager.cc:1428] T 10a780a47e63441895d181e08860d516 P 49ede907cb434995b3b28267e5a43baa: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
W20260810 00:45:58.705333 7758 tablet.cc:2369] T 10a780a47e63441895d181e08860d516 P e74119537bff4b4ab0ca81a2ecc85fc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:45:58.854889 8187 raft_consensus.cc:482] T 10a780a47e63441895d181e08860d516 P e74119537bff4b4ab0ca81a2ecc85fc8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:45:58.855172 8187 raft_consensus.cc:504] T 10a780a47e63441895d181e08860d516 P e74119537bff4b4ab0ca81a2ecc85fc8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } }
I20260810 00:45:58.855850 8187 leader_election.cc:288] T 10a780a47e63441895d181e08860d516 P e74119537bff4b4ab0ca81a2ecc85fc8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49ede907cb434995b3b28267e5a43baa (127.28.6.2:35007), 5e4f9e7de3134a81a712da7e20a37f20 (127.28.6.3:41251)
I20260810 00:45:58.856456 8082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10a780a47e63441895d181e08860d516" candidate_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e4f9e7de3134a81a712da7e20a37f20" is_pre_election: true
I20260810 00:45:58.856494 7880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10a780a47e63441895d181e08860d516" candidate_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49ede907cb434995b3b28267e5a43baa" is_pre_election: true
I20260810 00:45:58.856711 8082 raft_consensus.cc:2454] T 10a780a47e63441895d181e08860d516 P 5e4f9e7de3134a81a712da7e20a37f20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e74119537bff4b4ab0ca81a2ecc85fc8 in term 0.
I20260810 00:45:58.856784 7880 raft_consensus.cc:2454] T 10a780a47e63441895d181e08860d516 P 49ede907cb434995b3b28267e5a43baa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e74119537bff4b4ab0ca81a2ecc85fc8 in term 0.
I20260810 00:45:58.857229 7591 leader_election.cc:302] T 10a780a47e63441895d181e08860d516 P e74119537bff4b4ab0ca81a2ecc85fc8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e4f9e7de3134a81a712da7e20a37f20, e74119537bff4b4ab0ca81a2ecc85fc8; no voters:
I20260810 00:45:58.857558 8187 raft_consensus.cc:2781] T 10a780a47e63441895d181e08860d516 P e74119537bff4b4ab0ca81a2ecc85fc8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 00:45:58.857709 8187 raft_consensus.cc:482] T 10a780a47e63441895d181e08860d516 P e74119537bff4b4ab0ca81a2ecc85fc8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 00:45:58.857831 8187 raft_consensus.cc:3037] T 10a780a47e63441895d181e08860d516 P e74119537bff4b4ab0ca81a2ecc85fc8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:58.860431 8187 raft_consensus.cc:504] T 10a780a47e63441895d181e08860d516 P e74119537bff4b4ab0ca81a2ecc85fc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } }
I20260810 00:45:58.861213 8187 leader_election.cc:288] T 10a780a47e63441895d181e08860d516 P e74119537bff4b4ab0ca81a2ecc85fc8 [CANDIDATE]: Term 1 election: Requested vote from peers 49ede907cb434995b3b28267e5a43baa (127.28.6.2:35007), 5e4f9e7de3134a81a712da7e20a37f20 (127.28.6.3:41251)
I20260810 00:45:58.861555 7880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10a780a47e63441895d181e08860d516" candidate_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49ede907cb434995b3b28267e5a43baa"
I20260810 00:45:58.861788 7880 raft_consensus.cc:3037] T 10a780a47e63441895d181e08860d516 P 49ede907cb434995b3b28267e5a43baa [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:58.861732 8082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10a780a47e63441895d181e08860d516" candidate_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e4f9e7de3134a81a712da7e20a37f20"
I20260810 00:45:58.861954 8082 raft_consensus.cc:3037] T 10a780a47e63441895d181e08860d516 P 5e4f9e7de3134a81a712da7e20a37f20 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:58.863839 7880 raft_consensus.cc:2454] T 10a780a47e63441895d181e08860d516 P 49ede907cb434995b3b28267e5a43baa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e74119537bff4b4ab0ca81a2ecc85fc8 in term 1.
I20260810 00:45:58.864279 7592 leader_election.cc:302] T 10a780a47e63441895d181e08860d516 P e74119537bff4b4ab0ca81a2ecc85fc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49ede907cb434995b3b28267e5a43baa, e74119537bff4b4ab0ca81a2ecc85fc8; no voters:
I20260810 00:45:58.864594 8187 raft_consensus.cc:2781] T 10a780a47e63441895d181e08860d516 P e74119537bff4b4ab0ca81a2ecc85fc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:45:58.864809 8187 raft_consensus.cc:686] T 10a780a47e63441895d181e08860d516 P e74119537bff4b4ab0ca81a2ecc85fc8 [term 1 LEADER]: Becoming Leader. State: Replica: e74119537bff4b4ab0ca81a2ecc85fc8, State: Running, Role: LEADER
I20260810 00:45:58.864995 8082 raft_consensus.cc:2454] T 10a780a47e63441895d181e08860d516 P 5e4f9e7de3134a81a712da7e20a37f20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e74119537bff4b4ab0ca81a2ecc85fc8 in term 1.
I20260810 00:45:58.865089 8187 consensus_queue.cc:237] T 10a780a47e63441895d181e08860d516 P e74119537bff4b4ab0ca81a2ecc85fc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } }
I20260810 00:45:58.867892 7465 catalog_manager.cc:5495] T 10a780a47e63441895d181e08860d516 P e74119537bff4b4ab0ca81a2ecc85fc8 reported cstate change: term changed from 0 to 1, leader changed from <none> to e74119537bff4b4ab0ca81a2ecc85fc8 (127.28.6.1). New cstate: current_term: 1 leader_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } health_report { overall_health: UNKNOWN } } }
Aug 10 00:45:58 dist-test-slave-1nl6 krb5kdc[7401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322756, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.6.62@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM
I20260810 00:45:58.975476 8227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:58.953143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:45123 (local address 127.0.0.1:58150)
0810 00:45:58.955723 (+ 2580us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:58.955739 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:58.955947 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:58.956504 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:58.956513 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:58.957159 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:45:58.957368 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:58.957377 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:58.959081 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:58.959085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:58.960099 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:58.960111 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:58.961223 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:58.961227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:58.961360 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:58.966356 (+ 4996us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:45:58.966376 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:45:58.968660 (+ 2284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:45:58.972805 (+ 4145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:58.972817 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:58.972828 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:58.973346 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:58.973776 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:45:58.973779 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:45:58.973781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:45:58.973966 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:45:58.974616 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:45:58.974623 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:45:58.974766 (+ 143us) client_negotiation.cc:769] Sending connection context
0810 00:45:58.974960 (+ 194us) client_negotiation.cc:240] Negotiation successful
0810 00:45:58.975188 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2426,"thread_start_us":2250,"threads_started":1}
I20260810 00:45:58.975816 8228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:58.953302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58150 (local address 127.28.6.62:45123)
0810 00:45:58.955515 (+ 2213us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:58.955526 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:58.955875 (+ 349us) server_negotiation.cc:407] Connection header received
0810 00:45:58.956120 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:58.956126 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:58.956277 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:58.956427 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:58.957543 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:58.958878 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:58.960265 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:58.961042 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:58.961231 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:58.968890 (+ 7659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:58.968898 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:58.968902 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:58.968939 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:58.972672 (+ 3733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:58.973481 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:58.973487 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:58.973491 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:58.973660 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:58.974089 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:58.974096 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:58.974101 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:58.974487 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:58.974639 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:58.975068 (+ 429us) server_negotiation.cc:299] Negotiation successful
0810 00:45:58.975371 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":2012,"thread_start_us":1830,"threads_started":1}
I20260810 00:45:58.981088 7465 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:58150:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 00:45:58.982137 7465 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 00:45:58.989297 7657 tablet_service.cc:1467] Processing CreateTablet for tablet c73505342f004cbead39099d4ed8a4d6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8d88850fc1b04ced86734d8e95acbe91]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 00:45:58.989646 7853 tablet_service.cc:1467] Processing CreateTablet for tablet c73505342f004cbead39099d4ed8a4d6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8d88850fc1b04ced86734d8e95acbe91]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 00:45:58.989653 8049 tablet_service.cc:1467] Processing CreateTablet for tablet c73505342f004cbead39099d4ed8a4d6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8d88850fc1b04ced86734d8e95acbe91]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 00:45:58.989967 7657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c73505342f004cbead39099d4ed8a4d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:58.990218 7853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c73505342f004cbead39099d4ed8a4d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:58.990252 8049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c73505342f004cbead39099d4ed8a4d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:58.997910 8180 tablet_bootstrap.cc:492] T c73505342f004cbead39099d4ed8a4d6 P e74119537bff4b4ab0ca81a2ecc85fc8: Bootstrap starting.
I20260810 00:45:58.997929 8182 tablet_bootstrap.cc:492] T c73505342f004cbead39099d4ed8a4d6 P 5e4f9e7de3134a81a712da7e20a37f20: Bootstrap starting.
I20260810 00:45:58.999701 8182 tablet_bootstrap.cc:654] T c73505342f004cbead39099d4ed8a4d6 P 5e4f9e7de3134a81a712da7e20a37f20: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:58.999693 8180 tablet_bootstrap.cc:654] T c73505342f004cbead39099d4ed8a4d6 P e74119537bff4b4ab0ca81a2ecc85fc8: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:59.000851 8181 tablet_bootstrap.cc:492] T c73505342f004cbead39099d4ed8a4d6 P 49ede907cb434995b3b28267e5a43baa: Bootstrap starting.
I20260810 00:45:59.003155 8181 tablet_bootstrap.cc:654] T c73505342f004cbead39099d4ed8a4d6 P 49ede907cb434995b3b28267e5a43baa: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:59.012086 8180 tablet_bootstrap.cc:492] T c73505342f004cbead39099d4ed8a4d6 P e74119537bff4b4ab0ca81a2ecc85fc8: No bootstrap required, opened a new log
I20260810 00:45:59.012347 8180 ts_tablet_manager.cc:1397] T c73505342f004cbead39099d4ed8a4d6 P e74119537bff4b4ab0ca81a2ecc85fc8: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.003s
I20260810 00:45:59.013284 8182 tablet_bootstrap.cc:492] T c73505342f004cbead39099d4ed8a4d6 P 5e4f9e7de3134a81a712da7e20a37f20: No bootstrap required, opened a new log
I20260810 00:45:59.013502 8182 ts_tablet_manager.cc:1397] T c73505342f004cbead39099d4ed8a4d6 P 5e4f9e7de3134a81a712da7e20a37f20: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.003s
I20260810 00:45:59.013363 8180 raft_consensus.cc:348] T c73505342f004cbead39099d4ed8a4d6 P e74119537bff4b4ab0ca81a2ecc85fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } }
I20260810 00:45:59.013708 8180 raft_consensus.cc:374] T c73505342f004cbead39099d4ed8a4d6 P e74119537bff4b4ab0ca81a2ecc85fc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:59.013969 8180 raft_consensus.cc:729] T c73505342f004cbead39099d4ed8a4d6 P e74119537bff4b4ab0ca81a2ecc85fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e74119537bff4b4ab0ca81a2ecc85fc8, State: Initialized, Role: FOLLOWER
I20260810 00:45:59.014407 8180 consensus_queue.cc:260] T c73505342f004cbead39099d4ed8a4d6 P e74119537bff4b4ab0ca81a2ecc85fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } }
I20260810 00:45:59.015576 8182 raft_consensus.cc:348] T c73505342f004cbead39099d4ed8a4d6 P 5e4f9e7de3134a81a712da7e20a37f20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } }
I20260810 00:45:59.015875 8182 raft_consensus.cc:374] T c73505342f004cbead39099d4ed8a4d6 P 5e4f9e7de3134a81a712da7e20a37f20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:59.015985 8182 raft_consensus.cc:729] T c73505342f004cbead39099d4ed8a4d6 P 5e4f9e7de3134a81a712da7e20a37f20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e4f9e7de3134a81a712da7e20a37f20, State: Initialized, Role: FOLLOWER
I20260810 00:45:59.016286 8182 consensus_queue.cc:260] T c73505342f004cbead39099d4ed8a4d6 P 5e4f9e7de3134a81a712da7e20a37f20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } }
I20260810 00:45:59.014998 8187 tablet_replica.cc:439] T c73505342f004cbead39099d4ed8a4d6 P e74119537bff4b4ab0ca81a2ecc85fc8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } } }
I20260810 00:45:59.016564 8187 tablet_replica.cc:442] T c73505342f004cbead39099d4ed8a4d6 P e74119537bff4b4ab0ca81a2ecc85fc8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 00:45:59.017055 8182 ts_tablet_manager.cc:1428] T c73505342f004cbead39099d4ed8a4d6 P 5e4f9e7de3134a81a712da7e20a37f20: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260810 00:45:59.017118 8181 tablet_bootstrap.cc:492] T c73505342f004cbead39099d4ed8a4d6 P 49ede907cb434995b3b28267e5a43baa: No bootstrap required, opened a new log
I20260810 00:45:59.017586 8191 tablet_replica.cc:439] T c73505342f004cbead39099d4ed8a4d6 P 5e4f9e7de3134a81a712da7e20a37f20: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } } }
I20260810 00:45:59.017920 8191 tablet_replica.cc:442] T c73505342f004cbead39099d4ed8a4d6 P 5e4f9e7de3134a81a712da7e20a37f20: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 00:45:59.018249 8181 ts_tablet_manager.cc:1397] T c73505342f004cbead39099d4ed8a4d6 P 49ede907cb434995b3b28267e5a43baa: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.012s
I20260810 00:45:59.018787 8180 ts_tablet_manager.cc:1428] T c73505342f004cbead39099d4ed8a4d6 P e74119537bff4b4ab0ca81a2ecc85fc8: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20260810 00:45:59.019737 8181 raft_consensus.cc:348] T c73505342f004cbead39099d4ed8a4d6 P 49ede907cb434995b3b28267e5a43baa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } }
I20260810 00:45:59.020051 8181 raft_consensus.cc:374] T c73505342f004cbead39099d4ed8a4d6 P 49ede907cb434995b3b28267e5a43baa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:59.020176 8181 raft_consensus.cc:729] T c73505342f004cbead39099d4ed8a4d6 P 49ede907cb434995b3b28267e5a43baa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49ede907cb434995b3b28267e5a43baa, State: Initialized, Role: FOLLOWER
I20260810 00:45:59.020474 8181 consensus_queue.cc:260] T c73505342f004cbead39099d4ed8a4d6 P 49ede907cb434995b3b28267e5a43baa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } }
I20260810 00:45:59.020998 8190 tablet_replica.cc:439] T c73505342f004cbead39099d4ed8a4d6 P 49ede907cb434995b3b28267e5a43baa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } } }
I20260810 00:45:59.021306 8190 tablet_replica.cc:442] T c73505342f004cbead39099d4ed8a4d6 P 49ede907cb434995b3b28267e5a43baa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 00:45:59.021857 8181 ts_tablet_manager.cc:1428] T c73505342f004cbead39099d4ed8a4d6 P 49ede907cb434995b3b28267e5a43baa: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260810 00:45:59.121930 8187 raft_consensus.cc:482] T c73505342f004cbead39099d4ed8a4d6 P e74119537bff4b4ab0ca81a2ecc85fc8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:45:59.122186 8187 raft_consensus.cc:504] T c73505342f004cbead39099d4ed8a4d6 P e74119537bff4b4ab0ca81a2ecc85fc8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } }
I20260810 00:45:59.122888 8187 leader_election.cc:288] T c73505342f004cbead39099d4ed8a4d6 P e74119537bff4b4ab0ca81a2ecc85fc8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49ede907cb434995b3b28267e5a43baa (127.28.6.2:35007), 5e4f9e7de3134a81a712da7e20a37f20 (127.28.6.3:41251)
I20260810 00:45:59.123387 7880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73505342f004cbead39099d4ed8a4d6" candidate_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49ede907cb434995b3b28267e5a43baa" is_pre_election: true
I20260810 00:45:59.123487 8082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73505342f004cbead39099d4ed8a4d6" candidate_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e4f9e7de3134a81a712da7e20a37f20" is_pre_election: true
I20260810 00:45:59.123656 7880 raft_consensus.cc:2454] T c73505342f004cbead39099d4ed8a4d6 P 49ede907cb434995b3b28267e5a43baa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e74119537bff4b4ab0ca81a2ecc85fc8 in term 0.
I20260810 00:45:59.123828 8082 raft_consensus.cc:2454] T c73505342f004cbead39099d4ed8a4d6 P 5e4f9e7de3134a81a712da7e20a37f20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e74119537bff4b4ab0ca81a2ecc85fc8 in term 0.
I20260810 00:45:59.124195 7592 leader_election.cc:302] T c73505342f004cbead39099d4ed8a4d6 P e74119537bff4b4ab0ca81a2ecc85fc8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49ede907cb434995b3b28267e5a43baa, e74119537bff4b4ab0ca81a2ecc85fc8; no voters:
I20260810 00:45:59.124493 8187 raft_consensus.cc:2781] T c73505342f004cbead39099d4ed8a4d6 P e74119537bff4b4ab0ca81a2ecc85fc8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 00:45:59.124581 8187 raft_consensus.cc:482] T c73505342f004cbead39099d4ed8a4d6 P e74119537bff4b4ab0ca81a2ecc85fc8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 00:45:59.124701 8187 raft_consensus.cc:3037] T c73505342f004cbead39099d4ed8a4d6 P e74119537bff4b4ab0ca81a2ecc85fc8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:59.126920 8187 raft_consensus.cc:504] T c73505342f004cbead39099d4ed8a4d6 P e74119537bff4b4ab0ca81a2ecc85fc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } }
I20260810 00:45:59.127508 8187 leader_election.cc:288] T c73505342f004cbead39099d4ed8a4d6 P e74119537bff4b4ab0ca81a2ecc85fc8 [CANDIDATE]: Term 1 election: Requested vote from peers 49ede907cb434995b3b28267e5a43baa (127.28.6.2:35007), 5e4f9e7de3134a81a712da7e20a37f20 (127.28.6.3:41251)
I20260810 00:45:59.127956 7880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73505342f004cbead39099d4ed8a4d6" candidate_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49ede907cb434995b3b28267e5a43baa"
I20260810 00:45:59.127982 8082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73505342f004cbead39099d4ed8a4d6" candidate_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e4f9e7de3134a81a712da7e20a37f20"
I20260810 00:45:59.128163 7880 raft_consensus.cc:3037] T c73505342f004cbead39099d4ed8a4d6 P 49ede907cb434995b3b28267e5a43baa [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:59.128191 8082 raft_consensus.cc:3037] T c73505342f004cbead39099d4ed8a4d6 P 5e4f9e7de3134a81a712da7e20a37f20 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:59.130273 8082 raft_consensus.cc:2454] T c73505342f004cbead39099d4ed8a4d6 P 5e4f9e7de3134a81a712da7e20a37f20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e74119537bff4b4ab0ca81a2ecc85fc8 in term 1.
I20260810 00:45:59.130447 7880 raft_consensus.cc:2454] T c73505342f004cbead39099d4ed8a4d6 P 49ede907cb434995b3b28267e5a43baa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e74119537bff4b4ab0ca81a2ecc85fc8 in term 1.
I20260810 00:45:59.130707 7591 leader_election.cc:302] T c73505342f004cbead39099d4ed8a4d6 P e74119537bff4b4ab0ca81a2ecc85fc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e4f9e7de3134a81a712da7e20a37f20, e74119537bff4b4ab0ca81a2ecc85fc8; no voters:
I20260810 00:45:59.130996 8187 raft_consensus.cc:2781] T c73505342f004cbead39099d4ed8a4d6 P e74119537bff4b4ab0ca81a2ecc85fc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:45:59.131171 8187 raft_consensus.cc:686] T c73505342f004cbead39099d4ed8a4d6 P e74119537bff4b4ab0ca81a2ecc85fc8 [term 1 LEADER]: Becoming Leader. State: Replica: e74119537bff4b4ab0ca81a2ecc85fc8, State: Running, Role: LEADER
I20260810 00:45:59.131433 8187 consensus_queue.cc:237] T c73505342f004cbead39099d4ed8a4d6 P e74119537bff4b4ab0ca81a2ecc85fc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } }
I20260810 00:45:59.133049 8195 tablet_replica.cc:439] T c73505342f004cbead39099d4ed8a4d6 P e74119537bff4b4ab0ca81a2ecc85fc8: TxnStatusTablet state changed. Reason: New leader e74119537bff4b4ab0ca81a2ecc85fc8. Latest consensus state: current_term: 1 leader_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } } }
I20260810 00:45:59.133319 8195 tablet_replica.cc:442] T c73505342f004cbead39099d4ed8a4d6 P e74119537bff4b4ab0ca81a2ecc85fc8: This TxnStatusTablet replica's current role is: LEADER
I20260810 00:45:59.133946 8235 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 00:45:59.134189 8235 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 00:45:59.134706 7463 catalog_manager.cc:5495] T c73505342f004cbead39099d4ed8a4d6 P e74119537bff4b4ab0ca81a2ecc85fc8 reported cstate change: term changed from 0 to 1, leader changed from <none> to e74119537bff4b4ab0ca81a2ecc85fc8 (127.28.6.1). New cstate: current_term: 1 leader_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } health_report { overall_health: UNKNOWN } } }
I20260810 00:45:59.171080 8227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:59.164935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:44791 (local address 127.0.0.1:47092)
0810 00:45:59.165270 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:59.165296 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:59.165455 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:59.165947 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:59.165951 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:59.165962 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 00:45:59.166116 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:59.166124 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:59.167666 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:59.167670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:59.168654 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:59.168666 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:59.169668 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:59.169672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:59.169920 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:59.169975 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 00:45:59.170492 (+ 517us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 00:45:59.170509 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 00:45:59.170574 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 00:45:59.170711 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260810 00:45:59.171200 8237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:59.164920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47092 (local address 127.28.6.1:44791)
0810 00:45:59.165602 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:59.165609 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:59.165630 (+ 21us) server_negotiation.cc:407] Connection header received
0810 00:45:59.165704 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:59.165710 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:59.165795 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:59.165907 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:45:59.166284 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:59.167544 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:59.168794 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:59.169470 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:59.169591 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:59.170178 (+ 587us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:45:59.170382 (+ 204us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:45:59.170486 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:59.170656 (+ 170us) server_negotiation.cc:299] Negotiation successful
0810 00:45:59.170733 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":354,"threads_started":1}
I20260810 00:45:59.175640 7880 raft_consensus.cc:1261] T c73505342f004cbead39099d4ed8a4d6 P 49ede907cb434995b3b28267e5a43baa [term 1 FOLLOWER]: Refusing update from remote peer e74119537bff4b4ab0ca81a2ecc85fc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:45:59.175684 8082 raft_consensus.cc:1261] T c73505342f004cbead39099d4ed8a4d6 P 5e4f9e7de3134a81a712da7e20a37f20 [term 1 FOLLOWER]: Refusing update from remote peer e74119537bff4b4ab0ca81a2ecc85fc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:45:59.176343 8187 consensus_queue.cc:1035] T c73505342f004cbead39099d4ed8a4d6 P e74119537bff4b4ab0ca81a2ecc85fc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:45:59.176662 8195 consensus_queue.cc:1035] T c73505342f004cbead39099d4ed8a4d6 P e74119537bff4b4ab0ca81a2ecc85fc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:45:59.180294 8190 tablet_replica.cc:439] T c73505342f004cbead39099d4ed8a4d6 P 49ede907cb434995b3b28267e5a43baa: TxnStatusTablet state changed. Reason: New leader e74119537bff4b4ab0ca81a2ecc85fc8. Latest consensus state: current_term: 1 leader_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } } }
I20260810 00:45:59.180578 8190 tablet_replica.cc:442] T c73505342f004cbead39099d4ed8a4d6 P 49ede907cb434995b3b28267e5a43baa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 00:45:59.183563 8241 mvcc.cc:205] Tried to move back new op lower bound from 7316778021576704000 to 7316778021404078080. Current Snapshot: MvccSnapshot[applied={T|T < 7316778021576704000}]
I20260810 00:45:59.184027 8195 tablet_replica.cc:439] T c73505342f004cbead39099d4ed8a4d6 P e74119537bff4b4ab0ca81a2ecc85fc8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } } }
I20260810 00:45:59.184309 8195 tablet_replica.cc:442] T c73505342f004cbead39099d4ed8a4d6 P e74119537bff4b4ab0ca81a2ecc85fc8: This TxnStatusTablet replica's current role is: LEADER
I20260810 00:45:59.193025 8190 tablet_replica.cc:439] T c73505342f004cbead39099d4ed8a4d6 P 49ede907cb434995b3b28267e5a43baa: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } } }
I20260810 00:45:59.193288 8190 tablet_replica.cc:442] T c73505342f004cbead39099d4ed8a4d6 P 49ede907cb434995b3b28267e5a43baa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 00:45:59.195683 8191 tablet_replica.cc:439] T c73505342f004cbead39099d4ed8a4d6 P 5e4f9e7de3134a81a712da7e20a37f20: TxnStatusTablet state changed. Reason: New leader e74119537bff4b4ab0ca81a2ecc85fc8. Latest consensus state: current_term: 1 leader_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } } }
I20260810 00:45:59.195963 8191 tablet_replica.cc:442] T c73505342f004cbead39099d4ed8a4d6 P 5e4f9e7de3134a81a712da7e20a37f20: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 00:45:59.197371 8187 tablet_replica.cc:439] T c73505342f004cbead39099d4ed8a4d6 P e74119537bff4b4ab0ca81a2ecc85fc8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } } }
I20260810 00:45:59.197643 8187 tablet_replica.cc:442] T c73505342f004cbead39099d4ed8a4d6 P e74119537bff4b4ab0ca81a2ecc85fc8: This TxnStatusTablet replica's current role is: LEADER
I20260810 00:45:59.201040 8191 tablet_replica.cc:439] T c73505342f004cbead39099d4ed8a4d6 P 5e4f9e7de3134a81a712da7e20a37f20: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } } }
I20260810 00:45:59.201275 8191 tablet_replica.cc:442] T c73505342f004cbead39099d4ed8a4d6 P 5e4f9e7de3134a81a712da7e20a37f20: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 00:45:59.218992 8237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:59.211724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:35453 (local address 127.28.6.1:44791)
0810 00:45:59.212665 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:59.212674 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:59.212694 (+ 20us) server_negotiation.cc:407] Connection header received
0810 00:45:59.212935 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:59.212941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:59.213017 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:59.213120 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:45:59.213864 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:59.215110 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:59.216990 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:59.217614 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:59.217771 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:59.218196 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:45:59.218307 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:45:59.218412 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:59.218554 (+ 142us) server_negotiation.cc:299] Negotiation successful
0810 00:45:59.218624 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":824}
I20260810 00:45:59.219184 8250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:59.211740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:44791 (local address 127.28.6.1:35453)
0810 00:45:59.212602 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:59.212625 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:59.212793 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:59.213133 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:59.213138 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:59.213149 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 00:45:59.213686 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:59.213698 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:59.215270 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:59.215276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:59.216833 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:59.216847 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:59.217793 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:59.217800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:59.217948 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:59.218014 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 00:45:59.218412 (+ 398us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 00:45:59.218426 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 00:45:59.218533 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 00:45:59.218677 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":464,"threads_started":1}
I20260810 00:45:59.230268 8082 raft_consensus.cc:1261] T 10a780a47e63441895d181e08860d516 P 5e4f9e7de3134a81a712da7e20a37f20 [term 1 FOLLOWER]: Refusing update from remote peer e74119537bff4b4ab0ca81a2ecc85fc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:45:59.230434 7880 raft_consensus.cc:1261] T 10a780a47e63441895d181e08860d516 P 49ede907cb434995b3b28267e5a43baa [term 1 FOLLOWER]: Refusing update from remote peer e74119537bff4b4ab0ca81a2ecc85fc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:45:59.231026 8195 consensus_queue.cc:1035] T 10a780a47e63441895d181e08860d516 P e74119537bff4b4ab0ca81a2ecc85fc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:45:59.231320 8187 consensus_queue.cc:1035] T 10a780a47e63441895d181e08860d516 P e74119537bff4b4ab0ca81a2ecc85fc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:45:59.242681 8238 mvcc.cc:205] Tried to move back new op lower bound from 7316778021800607744 to 7316778020314071040. Current Snapshot: MvccSnapshot[applied={T|T < 7316778021800607744}]
I20260810 00:45:59.254356 8239 mvcc.cc:205] Tried to move back new op lower bound from 7316778021800607744 to 7316778020314071040. Current Snapshot: MvccSnapshot[applied={T|T < 7316778021800607744}]
I20260810 00:45:59.306083 7465 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:45:59.306366 7465 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:45:59.309787 7465 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 89582f0c9b9943f2ba432c32687a6435: Sending DeleteTablet for 3 replicas of tablet 10a780a47e63441895d181e08860d516
I20260810 00:45:59.310787 8049 tablet_service.cc:1514] Processing DeleteTablet for tablet 10a780a47e63441895d181e08860d516 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:45:59 UTC) from {username='oryx', principal='oryx/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:60896
I20260810 00:45:59.311121 7853 tablet_service.cc:1514] Processing DeleteTablet for tablet 10a780a47e63441895d181e08860d516 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:45:59 UTC) from {username='oryx', principal='oryx/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:53188
I20260810 00:45:59.313819 7657 tablet_service.cc:1514] Processing DeleteTablet for tablet 10a780a47e63441895d181e08860d516 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:45:59 UTC) from {username='oryx', principal='oryx/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:47078
I20260810 00:45:59.339913 8261 tablet_replica.cc:330] T 10a780a47e63441895d181e08860d516 P 49ede907cb434995b3b28267e5a43baa: stopping tablet replica
I20260810 00:45:59.340243 8261 raft_consensus.cc:2229] T 10a780a47e63441895d181e08860d516 P 49ede907cb434995b3b28267e5a43baa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:45:59.340564 8261 raft_consensus.cc:2258] T 10a780a47e63441895d181e08860d516 P 49ede907cb434995b3b28267e5a43baa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:45:59.341472 8260 tablet_replica.cc:330] T 10a780a47e63441895d181e08860d516 P 5e4f9e7de3134a81a712da7e20a37f20: stopping tablet replica
I20260810 00:45:59.341667 8260 raft_consensus.cc:2229] T 10a780a47e63441895d181e08860d516 P 5e4f9e7de3134a81a712da7e20a37f20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:45:59.341835 8261 ts_tablet_manager.cc:1905] T 10a780a47e63441895d181e08860d516 P 49ede907cb434995b3b28267e5a43baa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:45:59.341919 8260 raft_consensus.cc:2258] T 10a780a47e63441895d181e08860d516 P 5e4f9e7de3134a81a712da7e20a37f20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:45:59.342203 8262 tablet_replica.cc:330] T 10a780a47e63441895d181e08860d516 P e74119537bff4b4ab0ca81a2ecc85fc8: stopping tablet replica
I20260810 00:45:59.342406 8262 raft_consensus.cc:2229] T 10a780a47e63441895d181e08860d516 P e74119537bff4b4ab0ca81a2ecc85fc8 [term 1 LEADER]: Raft consensus shutting down.
I20260810 00:45:59.342872 8262 raft_consensus.cc:2258] T 10a780a47e63441895d181e08860d516 P e74119537bff4b4ab0ca81a2ecc85fc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:45:59.343672 8260 ts_tablet_manager.cc:1905] T 10a780a47e63441895d181e08860d516 P 5e4f9e7de3134a81a712da7e20a37f20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:45:59.343982 8262 ts_tablet_manager.cc:1905] T 10a780a47e63441895d181e08860d516 P e74119537bff4b4ab0ca81a2ecc85fc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:45:59.350850 8260 ts_tablet_manager.cc:1918] T 10a780a47e63441895d181e08860d516 P 5e4f9e7de3134a81a712da7e20a37f20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 00:45:59.351070 8260 log.cc:1196] T 10a780a47e63441895d181e08860d516 P 5e4f9e7de3134a81a712da7e20a37f20: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-2/wal/wals/10a780a47e63441895d181e08860d516
I20260810 00:45:59.351634 8260 ts_tablet_manager.cc:1939] T 10a780a47e63441895d181e08860d516 P 5e4f9e7de3134a81a712da7e20a37f20: Deleting consensus metadata
I20260810 00:45:59.351840 8262 ts_tablet_manager.cc:1918] T 10a780a47e63441895d181e08860d516 P e74119537bff4b4ab0ca81a2ecc85fc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 00:45:59.351979 8262 log.cc:1196] T 10a780a47e63441895d181e08860d516 P e74119537bff4b4ab0ca81a2ecc85fc8: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-0/wal/wals/10a780a47e63441895d181e08860d516
I20260810 00:45:59.352509 8262 ts_tablet_manager.cc:1939] T 10a780a47e63441895d181e08860d516 P e74119537bff4b4ab0ca81a2ecc85fc8: Deleting consensus metadata
I20260810 00:45:59.353312 7441 catalog_manager.cc:4841] TS 5e4f9e7de3134a81a712da7e20a37f20 (127.28.6.3:41251): tablet 10a780a47e63441895d181e08860d516 (table test-table [id=2705ea9efd0c414fb5d1768aebd9e212]) successfully deleted
I20260810 00:45:59.353662 7441 catalog_manager.cc:4841] TS e74119537bff4b4ab0ca81a2ecc85fc8 (127.28.6.1:44791): tablet 10a780a47e63441895d181e08860d516 (table test-table [id=2705ea9efd0c414fb5d1768aebd9e212]) successfully deleted
I20260810 00:45:59.360486 8261 ts_tablet_manager.cc:1918] T 10a780a47e63441895d181e08860d516 P 49ede907cb434995b3b28267e5a43baa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 00:45:59.360630 8261 log.cc:1196] T 10a780a47e63441895d181e08860d516 P 49ede907cb434995b3b28267e5a43baa: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-1/wal/wals/10a780a47e63441895d181e08860d516
I20260810 00:45:59.361109 8261 ts_tablet_manager.cc:1939] T 10a780a47e63441895d181e08860d516 P 49ede907cb434995b3b28267e5a43baa: Deleting consensus metadata
I20260810 00:45:59.362172 7442 catalog_manager.cc:4841] TS 49ede907cb434995b3b28267e5a43baa (127.28.6.2:35007): tablet 10a780a47e63441895d181e08860d516 (table test-table [id=2705ea9efd0c414fb5d1768aebd9e212]) successfully deleted
Aug 10 00:45:59 dist-test-slave-1nl6 krb5kdc[7401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 00:45:59 dist-test-slave-1nl6 krb5kdc[7401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:45:59 dist-test-slave-1nl6 krb5kdc[7401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 00:45:59.401840 8228 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:45:59.389072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58154 (local address 127.28.6.62:45123)
0810 00:45:59.389293 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:59.389300 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:59.389878 (+ 578us) server_negotiation.cc:407] Connection header received
0810 00:45:59.390047 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:59.390053 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:59.390138 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:59.390265 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:59.391990 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:59.393295 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:59.395121 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:59.395779 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:59.395900 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:59.401562 (+ 5662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58154: BlockingRecv error: recv got EOF from 127.0.0.1:58154 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 10 00:45:59 dist-test-slave-1nl6 krb5kdc[7401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM
I20260810 00:45:59.424335 8228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:59.405063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58158 (local address 127.28.6.62:45123)
0810 00:45:59.405369 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:59.405376 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:59.407412 (+ 2036us) server_negotiation.cc:407] Connection header received
0810 00:45:59.407541 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:59.407547 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:59.407645 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:59.407775 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:45:59.408974 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:59.410275 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:59.411565 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:59.412486 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:59.412609 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:59.417886 (+ 5277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:45:59.417893 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:45:59.417898 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:45:59.418008 (+ 110us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:45:59.421388 (+ 3380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:59.422096 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:59.422102 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:59.422105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:59.422267 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:45:59.422759 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:45:59.422763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:45:59.422765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:45:59.423149 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:45:59.423420 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:59.423726 (+ 306us) server_negotiation.cc:299] Negotiation successful
0810 00:45:59.423983 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 00:45:59.427837 7465 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 00:45:59.428483 7465 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 00:45:59.435928 7657 tablet_service.cc:1467] Processing CreateTablet for tablet f787548d79ed4bd78f66d1b7126e00b5 (DEFAULT_TABLE table=test-table [id=eee74a318ac540e58d1a91ffd0f920dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:59.436021 7853 tablet_service.cc:1467] Processing CreateTablet for tablet f787548d79ed4bd78f66d1b7126e00b5 (DEFAULT_TABLE table=test-table [id=eee74a318ac540e58d1a91ffd0f920dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:59.436369 7853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f787548d79ed4bd78f66d1b7126e00b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:59.436417 7657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f787548d79ed4bd78f66d1b7126e00b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:59.440986 8181 tablet_bootstrap.cc:492] T f787548d79ed4bd78f66d1b7126e00b5 P 49ede907cb434995b3b28267e5a43baa: Bootstrap starting.
I20260810 00:45:59.442338 8181 tablet_bootstrap.cc:654] T f787548d79ed4bd78f66d1b7126e00b5 P 49ede907cb434995b3b28267e5a43baa: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:59.443702 8049 tablet_service.cc:1467] Processing CreateTablet for tablet f787548d79ed4bd78f66d1b7126e00b5 (DEFAULT_TABLE table=test-table [id=eee74a318ac540e58d1a91ffd0f920dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:59.444149 8049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f787548d79ed4bd78f66d1b7126e00b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:59.450634 8182 tablet_bootstrap.cc:492] T f787548d79ed4bd78f66d1b7126e00b5 P 5e4f9e7de3134a81a712da7e20a37f20: Bootstrap starting.
I20260810 00:45:59.452072 8180 tablet_bootstrap.cc:492] T f787548d79ed4bd78f66d1b7126e00b5 P e74119537bff4b4ab0ca81a2ecc85fc8: Bootstrap starting.
I20260810 00:45:59.452375 8182 tablet_bootstrap.cc:654] T f787548d79ed4bd78f66d1b7126e00b5 P 5e4f9e7de3134a81a712da7e20a37f20: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:59.454036 8180 tablet_bootstrap.cc:654] T f787548d79ed4bd78f66d1b7126e00b5 P e74119537bff4b4ab0ca81a2ecc85fc8: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:59.454739 8181 tablet_bootstrap.cc:492] T f787548d79ed4bd78f66d1b7126e00b5 P 49ede907cb434995b3b28267e5a43baa: No bootstrap required, opened a new log
I20260810 00:45:59.454977 8181 ts_tablet_manager.cc:1397] T f787548d79ed4bd78f66d1b7126e00b5 P 49ede907cb434995b3b28267e5a43baa: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.007s
I20260810 00:45:59.456084 8181 raft_consensus.cc:348] T f787548d79ed4bd78f66d1b7126e00b5 P 49ede907cb434995b3b28267e5a43baa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } }
I20260810 00:45:59.456535 8181 raft_consensus.cc:374] T f787548d79ed4bd78f66d1b7126e00b5 P 49ede907cb434995b3b28267e5a43baa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:59.456708 8181 raft_consensus.cc:729] T f787548d79ed4bd78f66d1b7126e00b5 P 49ede907cb434995b3b28267e5a43baa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49ede907cb434995b3b28267e5a43baa, State: Initialized, Role: FOLLOWER
I20260810 00:45:59.457113 8181 consensus_queue.cc:260] T f787548d79ed4bd78f66d1b7126e00b5 P 49ede907cb434995b3b28267e5a43baa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } }
I20260810 00:45:59.459365 8181 ts_tablet_manager.cc:1428] T f787548d79ed4bd78f66d1b7126e00b5 P 49ede907cb434995b3b28267e5a43baa: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260810 00:45:59.461244 8180 tablet_bootstrap.cc:492] T f787548d79ed4bd78f66d1b7126e00b5 P e74119537bff4b4ab0ca81a2ecc85fc8: No bootstrap required, opened a new log
I20260810 00:45:59.461467 8180 ts_tablet_manager.cc:1397] T f787548d79ed4bd78f66d1b7126e00b5 P e74119537bff4b4ab0ca81a2ecc85fc8: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260810 00:45:59.462525 8182 tablet_bootstrap.cc:492] T f787548d79ed4bd78f66d1b7126e00b5 P 5e4f9e7de3134a81a712da7e20a37f20: No bootstrap required, opened a new log
I20260810 00:45:59.462674 8182 ts_tablet_manager.cc:1397] T f787548d79ed4bd78f66d1b7126e00b5 P 5e4f9e7de3134a81a712da7e20a37f20: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.001s
I20260810 00:45:59.462455 8180 raft_consensus.cc:348] T f787548d79ed4bd78f66d1b7126e00b5 P e74119537bff4b4ab0ca81a2ecc85fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } }
I20260810 00:45:59.463457 8180 raft_consensus.cc:374] T f787548d79ed4bd78f66d1b7126e00b5 P e74119537bff4b4ab0ca81a2ecc85fc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:59.463639 8180 raft_consensus.cc:729] T f787548d79ed4bd78f66d1b7126e00b5 P e74119537bff4b4ab0ca81a2ecc85fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e74119537bff4b4ab0ca81a2ecc85fc8, State: Initialized, Role: FOLLOWER
I20260810 00:45:59.463542 8182 raft_consensus.cc:348] T f787548d79ed4bd78f66d1b7126e00b5 P 5e4f9e7de3134a81a712da7e20a37f20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } }
I20260810 00:45:59.463783 8182 raft_consensus.cc:374] T f787548d79ed4bd78f66d1b7126e00b5 P 5e4f9e7de3134a81a712da7e20a37f20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:59.463886 8182 raft_consensus.cc:729] T f787548d79ed4bd78f66d1b7126e00b5 P 5e4f9e7de3134a81a712da7e20a37f20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e4f9e7de3134a81a712da7e20a37f20, State: Initialized, Role: FOLLOWER
I20260810 00:45:59.464084 8180 consensus_queue.cc:260] T f787548d79ed4bd78f66d1b7126e00b5 P e74119537bff4b4ab0ca81a2ecc85fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } }
I20260810 00:45:59.464139 8182 consensus_queue.cc:260] T f787548d79ed4bd78f66d1b7126e00b5 P 5e4f9e7de3134a81a712da7e20a37f20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } }
I20260810 00:45:59.464676 8180 ts_tablet_manager.cc:1428] T f787548d79ed4bd78f66d1b7126e00b5 P e74119537bff4b4ab0ca81a2ecc85fc8: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260810 00:45:59.465466 8182 ts_tablet_manager.cc:1428] T f787548d79ed4bd78f66d1b7126e00b5 P 5e4f9e7de3134a81a712da7e20a37f20: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260810 00:45:59.629118 8195 raft_consensus.cc:482] T f787548d79ed4bd78f66d1b7126e00b5 P e74119537bff4b4ab0ca81a2ecc85fc8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:45:59.629047 8190 raft_consensus.cc:482] T f787548d79ed4bd78f66d1b7126e00b5 P 49ede907cb434995b3b28267e5a43baa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:45:59.629374 8195 raft_consensus.cc:504] T f787548d79ed4bd78f66d1b7126e00b5 P e74119537bff4b4ab0ca81a2ecc85fc8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } }
I20260810 00:45:59.629374 8190 raft_consensus.cc:504] T f787548d79ed4bd78f66d1b7126e00b5 P 49ede907cb434995b3b28267e5a43baa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } }
I20260810 00:45:59.630174 8195 leader_election.cc:288] T f787548d79ed4bd78f66d1b7126e00b5 P e74119537bff4b4ab0ca81a2ecc85fc8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49ede907cb434995b3b28267e5a43baa (127.28.6.2:35007), 5e4f9e7de3134a81a712da7e20a37f20 (127.28.6.3:41251)
I20260810 00:45:59.630775 7879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f787548d79ed4bd78f66d1b7126e00b5" candidate_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49ede907cb434995b3b28267e5a43baa" is_pre_election: true
I20260810 00:45:59.630820 8081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f787548d79ed4bd78f66d1b7126e00b5" candidate_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e4f9e7de3134a81a712da7e20a37f20" is_pre_election: true
I20260810 00:45:59.631088 7879 raft_consensus.cc:2454] T f787548d79ed4bd78f66d1b7126e00b5 P 49ede907cb434995b3b28267e5a43baa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e74119537bff4b4ab0ca81a2ecc85fc8 in term 0.
I20260810 00:45:59.631098 8081 raft_consensus.cc:2454] T f787548d79ed4bd78f66d1b7126e00b5 P 5e4f9e7de3134a81a712da7e20a37f20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e74119537bff4b4ab0ca81a2ecc85fc8 in term 0.
I20260810 00:45:59.631323 8190 leader_election.cc:288] T f787548d79ed4bd78f66d1b7126e00b5 P 49ede907cb434995b3b28267e5a43baa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e74119537bff4b4ab0ca81a2ecc85fc8 (127.28.6.1:44791), 5e4f9e7de3134a81a712da7e20a37f20 (127.28.6.3:41251)
I20260810 00:45:59.631805 7591 leader_election.cc:302] T f787548d79ed4bd78f66d1b7126e00b5 P e74119537bff4b4ab0ca81a2ecc85fc8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e4f9e7de3134a81a712da7e20a37f20, e74119537bff4b4ab0ca81a2ecc85fc8; no voters:
I20260810 00:45:59.632120 8195 raft_consensus.cc:2781] T f787548d79ed4bd78f66d1b7126e00b5 P e74119537bff4b4ab0ca81a2ecc85fc8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 00:45:59.632238 8195 raft_consensus.cc:482] T f787548d79ed4bd78f66d1b7126e00b5 P e74119537bff4b4ab0ca81a2ecc85fc8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 00:45:59.632351 8195 raft_consensus.cc:3037] T f787548d79ed4bd78f66d1b7126e00b5 P e74119537bff4b4ab0ca81a2ecc85fc8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:59.635325 8195 raft_consensus.cc:504] T f787548d79ed4bd78f66d1b7126e00b5 P e74119537bff4b4ab0ca81a2ecc85fc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } }
I20260810 00:45:59.636310 8081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f787548d79ed4bd78f66d1b7126e00b5" candidate_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e4f9e7de3134a81a712da7e20a37f20"
I20260810 00:45:59.636343 7879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f787548d79ed4bd78f66d1b7126e00b5" candidate_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49ede907cb434995b3b28267e5a43baa"
I20260810 00:45:59.636512 8081 raft_consensus.cc:3037] T f787548d79ed4bd78f66d1b7126e00b5 P 5e4f9e7de3134a81a712da7e20a37f20 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:59.636541 7879 raft_consensus.cc:3037] T f787548d79ed4bd78f66d1b7126e00b5 P 49ede907cb434995b3b28267e5a43baa [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:59.639652 8081 raft_consensus.cc:2454] T f787548d79ed4bd78f66d1b7126e00b5 P 5e4f9e7de3134a81a712da7e20a37f20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e74119537bff4b4ab0ca81a2ecc85fc8 in term 1.
I20260810 00:45:59.639768 7879 raft_consensus.cc:2454] T f787548d79ed4bd78f66d1b7126e00b5 P 49ede907cb434995b3b28267e5a43baa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e74119537bff4b4ab0ca81a2ecc85fc8 in term 1.
I20260810 00:45:59.640096 7591 leader_election.cc:302] T f787548d79ed4bd78f66d1b7126e00b5 P e74119537bff4b4ab0ca81a2ecc85fc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e4f9e7de3134a81a712da7e20a37f20, e74119537bff4b4ab0ca81a2ecc85fc8; no voters:
I20260810 00:45:59.640411 8256 raft_consensus.cc:2781] T f787548d79ed4bd78f66d1b7126e00b5 P e74119537bff4b4ab0ca81a2ecc85fc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:45:59.640705 8237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:59.631584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:34701 (local address 127.28.6.1:44791)
0810 00:45:59.631784 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:59.631795 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:59.632272 (+ 477us) server_negotiation.cc:407] Connection header received
0810 00:45:59.632588 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:59.632595 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:59.632688 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:59.632756 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:59.634061 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:59.635361 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:59.638116 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:59.639418 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:59.639516 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:59.639625 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:59.640257 (+ 632us) server_negotiation.cc:299] Negotiation successful
0810 00:45:59.640415 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 00:45:59.640609 8256 raft_consensus.cc:686] T f787548d79ed4bd78f66d1b7126e00b5 P e74119537bff4b4ab0ca81a2ecc85fc8 [term 1 LEADER]: Becoming Leader. State: Replica: e74119537bff4b4ab0ca81a2ecc85fc8, State: Running, Role: LEADER
I20260810 00:45:59.640921 8286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:59.632354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:41251 (local address 127.28.6.2:45727)
0810 00:45:59.632943 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:59.632961 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:59.633081 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:59.633949 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:59.633956 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:59.633986 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:59.634373 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:59.634386 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:59.635902 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:59.635908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:59.638125 (+ 2217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:59.638138 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:59.640087 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:59.640092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:59.640271 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:59.640300 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 00:45:59.640388 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 00:45:59.640508 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":1,"thread_start_us":270,"threads_started":1}
I20260810 00:45:59.641002 8256 consensus_queue.cc:237] T f787548d79ed4bd78f66d1b7126e00b5 P e74119537bff4b4ab0ca81a2ecc85fc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } }
I20260810 00:45:59.641320 8285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:59.631529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:44791 (local address 127.28.6.2:34701)
0810 00:45:59.632177 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:59.632198 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:59.632385 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:59.633464 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:59.633468 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:59.633516 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:59.633867 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:59.633882 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:59.635545 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:59.635550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:59.637916 (+ 2366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:59.637932 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:59.639811 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:59.639818 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:59.640043 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:59.640088 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 00:45:59.640900 (+ 812us) client_negotiation.cc:240] Negotiation successful
0810 00:45:59.641039 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":401,"threads_started":1}
I20260810 00:45:59.641582 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:59.633039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:45727 (local address 127.28.6.3:41251)
0810 00:45:59.633501 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:59.633507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:59.633524 (+ 17us) server_negotiation.cc:407] Connection header received
0810 00:45:59.633587 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:59.633592 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:59.633662 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:59.633763 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:59.634594 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:59.635732 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:59.638303 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:59.639685 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:59.639803 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:59.639932 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:59.641102 (+ 1170us) server_negotiation.cc:299] Negotiation successful
0810 00:45:59.641246 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":239,"threads_started":1}
I20260810 00:45:59.641646 8195 leader_election.cc:288] T f787548d79ed4bd78f66d1b7126e00b5 P e74119537bff4b4ab0ca81a2ecc85fc8 [CANDIDATE]: Term 1 election: Requested vote from peers 49ede907cb434995b3b28267e5a43baa (127.28.6.2:35007), 5e4f9e7de3134a81a712da7e20a37f20 (127.28.6.3:41251)
I20260810 00:45:59.642087 8081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f787548d79ed4bd78f66d1b7126e00b5" candidate_uuid: "49ede907cb434995b3b28267e5a43baa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e4f9e7de3134a81a712da7e20a37f20" is_pre_election: true
I20260810 00:45:59.642443 8081 raft_consensus.cc:2379] T f787548d79ed4bd78f66d1b7126e00b5 P 5e4f9e7de3134a81a712da7e20a37f20 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 49ede907cb434995b3b28267e5a43baa in current term 1: Already voted for candidate e74119537bff4b4ab0ca81a2ecc85fc8 in this term.
I20260810 00:45:59.642521 7694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f787548d79ed4bd78f66d1b7126e00b5" candidate_uuid: "49ede907cb434995b3b28267e5a43baa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" is_pre_election: true
I20260810 00:45:59.643606 7784 leader_election.cc:302] T f787548d79ed4bd78f66d1b7126e00b5 P 49ede907cb434995b3b28267e5a43baa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49ede907cb434995b3b28267e5a43baa; no voters: 5e4f9e7de3134a81a712da7e20a37f20, e74119537bff4b4ab0ca81a2ecc85fc8
I20260810 00:45:59.643993 8190 raft_consensus.cc:2726] T f787548d79ed4bd78f66d1b7126e00b5 P 49ede907cb434995b3b28267e5a43baa [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 00:45:59.644994 7465 catalog_manager.cc:5495] T f787548d79ed4bd78f66d1b7126e00b5 P e74119537bff4b4ab0ca81a2ecc85fc8 reported cstate change: term changed from 0 to 1, leader changed from <none> to e74119537bff4b4ab0ca81a2ecc85fc8 (127.28.6.1). New cstate: current_term: 1 leader_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } health_report { overall_health: UNKNOWN } } }
I20260810 00:45:59.690290 8237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:59.685722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47094 (local address 127.28.6.1:44791)
0810 00:45:59.685918 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:59.685922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:59.685944 (+ 22us) server_negotiation.cc:407] Connection header received
0810 00:45:59.686142 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:59.686146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:59.686207 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:59.686280 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:45:59.686626 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:59.687425 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:59.688736 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:59.689141 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:59.689231 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:59.689618 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:45:59.689716 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:45:59.689782 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:59.689922 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 00:45:59.689978 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 00:45:59.692854 8081 raft_consensus.cc:1261] T f787548d79ed4bd78f66d1b7126e00b5 P 5e4f9e7de3134a81a712da7e20a37f20 [term 1 FOLLOWER]: Refusing update from remote peer e74119537bff4b4ab0ca81a2ecc85fc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:45:59.693154 7879 raft_consensus.cc:1261] T f787548d79ed4bd78f66d1b7126e00b5 P 49ede907cb434995b3b28267e5a43baa [term 1 FOLLOWER]: Refusing update from remote peer e74119537bff4b4ab0ca81a2ecc85fc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:45:59.693468 8195 consensus_queue.cc:1035] T f787548d79ed4bd78f66d1b7126e00b5 P e74119537bff4b4ab0ca81a2ecc85fc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:45:59.693842 8256 consensus_queue.cc:1035] T f787548d79ed4bd78f66d1b7126e00b5 P e74119537bff4b4ab0ca81a2ecc85fc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:45:59.706389 7462 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:45:59.706621 7462 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:45:59.710966 7462 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 89582f0c9b9943f2ba432c32687a6435: Sending DeleteTablet for 3 replicas of tablet f787548d79ed4bd78f66d1b7126e00b5
I20260810 00:45:59.711786 8049 tablet_service.cc:1514] Processing DeleteTablet for tablet f787548d79ed4bd78f66d1b7126e00b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:45:59 UTC) from {username='oryx', principal='oryx/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:60896
I20260810 00:45:59.712138 7657 tablet_service.cc:1514] Processing DeleteTablet for tablet f787548d79ed4bd78f66d1b7126e00b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:45:59 UTC) from {username='oryx', principal='oryx/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:47078
I20260810 00:45:59.712296 7853 tablet_service.cc:1514] Processing DeleteTablet for tablet f787548d79ed4bd78f66d1b7126e00b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:45:59 UTC) from {username='oryx', principal='oryx/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:53188
I20260810 00:45:59.712442 8260 tablet_replica.cc:330] T f787548d79ed4bd78f66d1b7126e00b5 P 5e4f9e7de3134a81a712da7e20a37f20: stopping tablet replica
I20260810 00:45:59.712598 8261 tablet_replica.cc:330] T f787548d79ed4bd78f66d1b7126e00b5 P 49ede907cb434995b3b28267e5a43baa: stopping tablet replica
I20260810 00:45:59.712643 8260 raft_consensus.cc:2229] T f787548d79ed4bd78f66d1b7126e00b5 P 5e4f9e7de3134a81a712da7e20a37f20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:45:59.712837 8261 raft_consensus.cc:2229] T f787548d79ed4bd78f66d1b7126e00b5 P 49ede907cb434995b3b28267e5a43baa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:45:59.712895 8260 raft_consensus.cc:2258] T f787548d79ed4bd78f66d1b7126e00b5 P 5e4f9e7de3134a81a712da7e20a37f20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:45:59.713088 8261 raft_consensus.cc:2258] T f787548d79ed4bd78f66d1b7126e00b5 P 49ede907cb434995b3b28267e5a43baa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:45:59.713176 8262 tablet_replica.cc:330] T f787548d79ed4bd78f66d1b7126e00b5 P e74119537bff4b4ab0ca81a2ecc85fc8: stopping tablet replica
I20260810 00:45:59.713370 8262 raft_consensus.cc:2229] T f787548d79ed4bd78f66d1b7126e00b5 P e74119537bff4b4ab0ca81a2ecc85fc8 [term 1 LEADER]: Raft consensus shutting down.
I20260810 00:45:59.713313 7462 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 00:45:59.713686 8262 raft_consensus.cc:2258] T f787548d79ed4bd78f66d1b7126e00b5 P e74119537bff4b4ab0ca81a2ecc85fc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:45:59.714294 8261 ts_tablet_manager.cc:1905] T f787548d79ed4bd78f66d1b7126e00b5 P 49ede907cb434995b3b28267e5a43baa: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 00:45:59.714315 7462 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 00:45:59.714566 8049 tablet_service.cc:1514] Processing DeleteTablet for tablet f787548d79ed4bd78f66d1b7126e00b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:45:59 UTC) from {username='oryx', principal='oryx/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:60896
I20260810 00:45:59.715490 8260 ts_tablet_manager.cc:1905] T f787548d79ed4bd78f66d1b7126e00b5 P 5e4f9e7de3134a81a712da7e20a37f20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:45:59.717947 8262 ts_tablet_manager.cc:1905] T f787548d79ed4bd78f66d1b7126e00b5 P e74119537bff4b4ab0ca81a2ecc85fc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:45:59.721216 8261 ts_tablet_manager.cc:1918] T f787548d79ed4bd78f66d1b7126e00b5 P 49ede907cb434995b3b28267e5a43baa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:45:59.721375 8261 log.cc:1196] T f787548d79ed4bd78f66d1b7126e00b5 P 49ede907cb434995b3b28267e5a43baa: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-1/wal/wals/f787548d79ed4bd78f66d1b7126e00b5
I20260810 00:45:59.721813 8261 ts_tablet_manager.cc:1939] T f787548d79ed4bd78f66d1b7126e00b5 P 49ede907cb434995b3b28267e5a43baa: Deleting consensus metadata
I20260810 00:45:59.723521 7442 catalog_manager.cc:4841] TS 49ede907cb434995b3b28267e5a43baa (127.28.6.2:35007): tablet f787548d79ed4bd78f66d1b7126e00b5 (table test-table [id=eee74a318ac540e58d1a91ffd0f920dd]) successfully deleted
I20260810 00:45:59.724555 8260 ts_tablet_manager.cc:1918] T f787548d79ed4bd78f66d1b7126e00b5 P 5e4f9e7de3134a81a712da7e20a37f20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:45:59.724673 8260 log.cc:1196] T f787548d79ed4bd78f66d1b7126e00b5 P 5e4f9e7de3134a81a712da7e20a37f20: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-2/wal/wals/f787548d79ed4bd78f66d1b7126e00b5
I20260810 00:45:59.725070 8260 ts_tablet_manager.cc:1939] T f787548d79ed4bd78f66d1b7126e00b5 P 5e4f9e7de3134a81a712da7e20a37f20: Deleting consensus metadata
I20260810 00:45:59.725339 8262 ts_tablet_manager.cc:1918] T f787548d79ed4bd78f66d1b7126e00b5 P e74119537bff4b4ab0ca81a2ecc85fc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:45:59.725484 8262 log.cc:1196] T f787548d79ed4bd78f66d1b7126e00b5 P e74119537bff4b4ab0ca81a2ecc85fc8: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-0/wal/wals/f787548d79ed4bd78f66d1b7126e00b5
I20260810 00:45:59.725909 8262 ts_tablet_manager.cc:1939] T f787548d79ed4bd78f66d1b7126e00b5 P e74119537bff4b4ab0ca81a2ecc85fc8: Deleting consensus metadata
I20260810 00:45:59.726454 8049 tablet_service.cc:1467] Processing CreateTablet for tablet 906f3feeda914f78b6028dd5e70ddb2d (DEFAULT_TABLE table=test-table [id=9ec6023eed644a5eb3c2c81e4b95246a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:59.726610 7441 catalog_manager.cc:4841] TS 5e4f9e7de3134a81a712da7e20a37f20 (127.28.6.3:41251): tablet f787548d79ed4bd78f66d1b7126e00b5 (table test-table [id=eee74a318ac540e58d1a91ffd0f920dd]) successfully deleted
I20260810 00:45:59.726791 8049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 906f3feeda914f78b6028dd5e70ddb2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:59.727317 7441 catalog_manager.cc:4841] TS e74119537bff4b4ab0ca81a2ecc85fc8 (127.28.6.1:44791): tablet f787548d79ed4bd78f66d1b7126e00b5 (table test-table [id=eee74a318ac540e58d1a91ffd0f920dd]) successfully deleted
I20260810 00:45:59.727417 7657 tablet_service.cc:1467] Processing CreateTablet for tablet 906f3feeda914f78b6028dd5e70ddb2d (DEFAULT_TABLE table=test-table [id=9ec6023eed644a5eb3c2c81e4b95246a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:59.727792 7657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 906f3feeda914f78b6028dd5e70ddb2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:59.727885 7853 tablet_service.cc:1467] Processing CreateTablet for tablet 906f3feeda914f78b6028dd5e70ddb2d (DEFAULT_TABLE table=test-table [id=9ec6023eed644a5eb3c2c81e4b95246a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:45:59.728266 7853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 906f3feeda914f78b6028dd5e70ddb2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:45:59.731875 8182 tablet_bootstrap.cc:492] T 906f3feeda914f78b6028dd5e70ddb2d P 5e4f9e7de3134a81a712da7e20a37f20: Bootstrap starting.
W20260810 00:45:59.733014 7441 catalog_manager.cc:4808] TS 5e4f9e7de3134a81a712da7e20a37f20 (127.28.6.3:41251): delete failed for tablet f787548d79ed4bd78f66d1b7126e00b5 because the tablet was not found. No further retry: Not found: Tablet not found: f787548d79ed4bd78f66d1b7126e00b5
I20260810 00:45:59.733592 8182 tablet_bootstrap.cc:654] T 906f3feeda914f78b6028dd5e70ddb2d P 5e4f9e7de3134a81a712da7e20a37f20: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:59.733745 8180 tablet_bootstrap.cc:492] T 906f3feeda914f78b6028dd5e70ddb2d P e74119537bff4b4ab0ca81a2ecc85fc8: Bootstrap starting.
I20260810 00:45:59.734301 8181 tablet_bootstrap.cc:492] T 906f3feeda914f78b6028dd5e70ddb2d P 49ede907cb434995b3b28267e5a43baa: Bootstrap starting.
I20260810 00:45:59.735446 8180 tablet_bootstrap.cc:654] T 906f3feeda914f78b6028dd5e70ddb2d P e74119537bff4b4ab0ca81a2ecc85fc8: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:59.735927 8181 tablet_bootstrap.cc:654] T 906f3feeda914f78b6028dd5e70ddb2d P 49ede907cb434995b3b28267e5a43baa: Neither blocks nor log segments found. Creating new log.
I20260810 00:45:59.737629 8180 tablet_bootstrap.cc:492] T 906f3feeda914f78b6028dd5e70ddb2d P e74119537bff4b4ab0ca81a2ecc85fc8: No bootstrap required, opened a new log
I20260810 00:45:59.737833 8180 ts_tablet_manager.cc:1397] T 906f3feeda914f78b6028dd5e70ddb2d P e74119537bff4b4ab0ca81a2ecc85fc8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260810 00:45:59.737995 8182 tablet_bootstrap.cc:492] T 906f3feeda914f78b6028dd5e70ddb2d P 5e4f9e7de3134a81a712da7e20a37f20: No bootstrap required, opened a new log
I20260810 00:45:59.738198 8182 ts_tablet_manager.cc:1397] T 906f3feeda914f78b6028dd5e70ddb2d P 5e4f9e7de3134a81a712da7e20a37f20: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260810 00:45:59.738569 8180 raft_consensus.cc:348] T 906f3feeda914f78b6028dd5e70ddb2d P e74119537bff4b4ab0ca81a2ecc85fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } } peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } }
I20260810 00:45:59.738803 8180 raft_consensus.cc:374] T 906f3feeda914f78b6028dd5e70ddb2d P e74119537bff4b4ab0ca81a2ecc85fc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:59.738864 8180 raft_consensus.cc:729] T 906f3feeda914f78b6028dd5e70ddb2d P e74119537bff4b4ab0ca81a2ecc85fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e74119537bff4b4ab0ca81a2ecc85fc8, State: Initialized, Role: FOLLOWER
I20260810 00:45:59.739153 8180 consensus_queue.cc:260] T 906f3feeda914f78b6028dd5e70ddb2d P e74119537bff4b4ab0ca81a2ecc85fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } } peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } }
I20260810 00:45:59.739180 8182 raft_consensus.cc:348] T 906f3feeda914f78b6028dd5e70ddb2d P 5e4f9e7de3134a81a712da7e20a37f20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } } peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } }
I20260810 00:45:59.739500 8182 raft_consensus.cc:374] T 906f3feeda914f78b6028dd5e70ddb2d P 5e4f9e7de3134a81a712da7e20a37f20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:59.739583 8182 raft_consensus.cc:729] T 906f3feeda914f78b6028dd5e70ddb2d P 5e4f9e7de3134a81a712da7e20a37f20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e4f9e7de3134a81a712da7e20a37f20, State: Initialized, Role: FOLLOWER
I20260810 00:45:59.739616 8180 ts_tablet_manager.cc:1428] T 906f3feeda914f78b6028dd5e70ddb2d P e74119537bff4b4ab0ca81a2ecc85fc8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 00:45:59.739892 8182 consensus_queue.cc:260] T 906f3feeda914f78b6028dd5e70ddb2d P 5e4f9e7de3134a81a712da7e20a37f20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } } peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } }
I20260810 00:45:59.743427 8182 ts_tablet_manager.cc:1428] T 906f3feeda914f78b6028dd5e70ddb2d P 5e4f9e7de3134a81a712da7e20a37f20: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260810 00:45:59.750434 8181 tablet_bootstrap.cc:492] T 906f3feeda914f78b6028dd5e70ddb2d P 49ede907cb434995b3b28267e5a43baa: No bootstrap required, opened a new log
I20260810 00:45:59.750619 8181 ts_tablet_manager.cc:1397] T 906f3feeda914f78b6028dd5e70ddb2d P 49ede907cb434995b3b28267e5a43baa: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.013s
I20260810 00:45:59.751356 8181 raft_consensus.cc:348] T 906f3feeda914f78b6028dd5e70ddb2d P 49ede907cb434995b3b28267e5a43baa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } } peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } }
I20260810 00:45:59.751598 8181 raft_consensus.cc:374] T 906f3feeda914f78b6028dd5e70ddb2d P 49ede907cb434995b3b28267e5a43baa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:45:59.751721 8181 raft_consensus.cc:729] T 906f3feeda914f78b6028dd5e70ddb2d P 49ede907cb434995b3b28267e5a43baa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49ede907cb434995b3b28267e5a43baa, State: Initialized, Role: FOLLOWER
I20260810 00:45:59.751981 8181 consensus_queue.cc:260] T 906f3feeda914f78b6028dd5e70ddb2d P 49ede907cb434995b3b28267e5a43baa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } } peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } }
I20260810 00:45:59.752466 8181 ts_tablet_manager.cc:1428] T 906f3feeda914f78b6028dd5e70ddb2d P 49ede907cb434995b3b28267e5a43baa: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 00:45:59.950588 8191 raft_consensus.cc:482] T 906f3feeda914f78b6028dd5e70ddb2d P 5e4f9e7de3134a81a712da7e20a37f20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:45:59.950831 8191 raft_consensus.cc:504] T 906f3feeda914f78b6028dd5e70ddb2d P 5e4f9e7de3134a81a712da7e20a37f20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } } peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } }
I20260810 00:45:59.952509 8191 leader_election.cc:288] T 906f3feeda914f78b6028dd5e70ddb2d P 5e4f9e7de3134a81a712da7e20a37f20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e74119537bff4b4ab0ca81a2ecc85fc8 (127.28.6.1:44791), 49ede907cb434995b3b28267e5a43baa (127.28.6.2:35007)
I20260810 00:45:59.960181 8237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:59.952797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:53699 (local address 127.28.6.1:44791)
0810 00:45:59.953042 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:59.953052 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:59.953786 (+ 734us) server_negotiation.cc:407] Connection header received
0810 00:45:59.954117 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:59.954125 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:59.954205 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:59.954276 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:59.955207 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:59.955964 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:59.957930 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:59.959022 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:59.959120 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:59.959211 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:59.959729 (+ 518us) server_negotiation.cc:299] Negotiation successful
0810 00:45:59.959866 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 00:45:59.960278 8301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:59.952672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:44791 (local address 127.28.6.3:53699)
0810 00:45:59.953642 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:59.953672 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:59.953914 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:59.954492 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:59.954496 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:59.954526 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:59.955032 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:59.955047 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:59.956106 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:59.956111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:59.957770 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:59.957786 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:59.959318 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:59.959327 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:59.959546 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:59.959590 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 00:45:59.959674 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 00:45:59.959794 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"mutex_wait_us":91,"thread_start_us":602,"threads_started":1}
I20260810 00:45:59.960362 8302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:59.952807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:35007 (local address 127.28.6.3:59837)
0810 00:45:59.954058 (+ 1251us) negotiation.cc:107] Waiting for socket to connect
0810 00:45:59.954078 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 00:45:59.954187 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:45:59.954608 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:45:59.954611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:45:59.954642 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:45:59.955188 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:59.955202 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:59.956373 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:59.956377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:59.957803 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:45:59.957816 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:45:59.959537 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:45:59.959542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:45:59.959673 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:59.959707 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 00:45:59.959799 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 00:45:59.959927 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1114,"mutex_wait_us":20,"thread_start_us":888,"threads_started":1}
I20260810 00:45:59.960367 8303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:45:59.953027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:59837 (local address 127.28.6.2:35007)
0810 00:45:59.954287 (+ 1260us) server_negotiation.cc:206] Beginning negotiation
0810 00:45:59.954293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:45:59.954313 (+ 20us) server_negotiation.cc:407] Connection header received
0810 00:45:59.954383 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:45:59.954389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:45:59.954481 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:45:59.954583 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:45:59.955398 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:59.956235 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:59.957996 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:45:59.959290 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:45:59.959420 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:45:59.959552 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 00:45:59.959855 (+ 303us) server_negotiation.cc:299] Negotiation successful
0810 00:45:59.959993 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1093,"thread_start_us":940,"threads_started":1}
I20260810 00:45:59.960958 7879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "906f3feeda914f78b6028dd5e70ddb2d" candidate_uuid: "5e4f9e7de3134a81a712da7e20a37f20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49ede907cb434995b3b28267e5a43baa" is_pre_election: true
I20260810 00:45:59.960990 7694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "906f3feeda914f78b6028dd5e70ddb2d" candidate_uuid: "5e4f9e7de3134a81a712da7e20a37f20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" is_pre_election: true
I20260810 00:45:59.961193 7879 raft_consensus.cc:2454] T 906f3feeda914f78b6028dd5e70ddb2d P 49ede907cb434995b3b28267e5a43baa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e4f9e7de3134a81a712da7e20a37f20 in term 0.
I20260810 00:45:59.961259 7694 raft_consensus.cc:2454] T 906f3feeda914f78b6028dd5e70ddb2d P e74119537bff4b4ab0ca81a2ecc85fc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e4f9e7de3134a81a712da7e20a37f20 in term 0.
I20260810 00:45:59.961683 7982 leader_election.cc:302] T 906f3feeda914f78b6028dd5e70ddb2d P 5e4f9e7de3134a81a712da7e20a37f20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 49ede907cb434995b3b28267e5a43baa, 5e4f9e7de3134a81a712da7e20a37f20, e74119537bff4b4ab0ca81a2ecc85fc8; no voters:
I20260810 00:45:59.962112 8191 raft_consensus.cc:2781] T 906f3feeda914f78b6028dd5e70ddb2d P 5e4f9e7de3134a81a712da7e20a37f20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 00:45:59.962260 8191 raft_consensus.cc:482] T 906f3feeda914f78b6028dd5e70ddb2d P 5e4f9e7de3134a81a712da7e20a37f20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 00:45:59.962371 8191 raft_consensus.cc:3037] T 906f3feeda914f78b6028dd5e70ddb2d P 5e4f9e7de3134a81a712da7e20a37f20 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:59.964582 8191 raft_consensus.cc:504] T 906f3feeda914f78b6028dd5e70ddb2d P 5e4f9e7de3134a81a712da7e20a37f20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } } peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } }
I20260810 00:45:59.965127 8191 leader_election.cc:288] T 906f3feeda914f78b6028dd5e70ddb2d P 5e4f9e7de3134a81a712da7e20a37f20 [CANDIDATE]: Term 1 election: Requested vote from peers e74119537bff4b4ab0ca81a2ecc85fc8 (127.28.6.1:44791), 49ede907cb434995b3b28267e5a43baa (127.28.6.2:35007)
I20260810 00:45:59.965605 7694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "906f3feeda914f78b6028dd5e70ddb2d" candidate_uuid: "5e4f9e7de3134a81a712da7e20a37f20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8"
I20260810 00:45:59.965619 7879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "906f3feeda914f78b6028dd5e70ddb2d" candidate_uuid: "5e4f9e7de3134a81a712da7e20a37f20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49ede907cb434995b3b28267e5a43baa"
I20260810 00:45:59.965798 7879 raft_consensus.cc:3037] T 906f3feeda914f78b6028dd5e70ddb2d P 49ede907cb434995b3b28267e5a43baa [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:59.965834 7694 raft_consensus.cc:3037] T 906f3feeda914f78b6028dd5e70ddb2d P e74119537bff4b4ab0ca81a2ecc85fc8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:45:59.967859 7879 raft_consensus.cc:2454] T 906f3feeda914f78b6028dd5e70ddb2d P 49ede907cb434995b3b28267e5a43baa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e4f9e7de3134a81a712da7e20a37f20 in term 1.
I20260810 00:45:59.968240 7982 leader_election.cc:302] T 906f3feeda914f78b6028dd5e70ddb2d P 5e4f9e7de3134a81a712da7e20a37f20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49ede907cb434995b3b28267e5a43baa, 5e4f9e7de3134a81a712da7e20a37f20; no voters:
I20260810 00:45:59.968479 7694 raft_consensus.cc:2454] T 906f3feeda914f78b6028dd5e70ddb2d P e74119537bff4b4ab0ca81a2ecc85fc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e4f9e7de3134a81a712da7e20a37f20 in term 1.
I20260810 00:45:59.968549 8191 raft_consensus.cc:2781] T 906f3feeda914f78b6028dd5e70ddb2d P 5e4f9e7de3134a81a712da7e20a37f20 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:45:59.969069 8191 raft_consensus.cc:686] T 906f3feeda914f78b6028dd5e70ddb2d P 5e4f9e7de3134a81a712da7e20a37f20 [term 1 LEADER]: Becoming Leader. State: Replica: 5e4f9e7de3134a81a712da7e20a37f20, State: Running, Role: LEADER
I20260810 00:45:59.969452 8191 consensus_queue.cc:237] T 906f3feeda914f78b6028dd5e70ddb2d P 5e4f9e7de3134a81a712da7e20a37f20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } } peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } }
I20260810 00:45:59.972884 7462 catalog_manager.cc:5495] T 906f3feeda914f78b6028dd5e70ddb2d P 5e4f9e7de3134a81a712da7e20a37f20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e4f9e7de3134a81a712da7e20a37f20 (127.28.6.3). New cstate: current_term: 1 leader_uuid: "5e4f9e7de3134a81a712da7e20a37f20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e4f9e7de3134a81a712da7e20a37f20" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 41251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 } health_report { overall_health: UNKNOWN } } }
I20260810 00:46:00.045977 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:00.041079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60908 (local address 127.28.6.3:41251)
0810 00:46:00.041246 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:00.041251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:00.041267 (+ 16us) server_negotiation.cc:407] Connection header received
0810 00:46:00.041429 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:00.041433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:00.041506 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:00.041633 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:46:00.041971 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:00.042712 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:00.044012 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:00.044428 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:00.044527 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:00.044908 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:46:00.045460 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:46:00.045569 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:00.045720 (+ 151us) server_negotiation.cc:299] Negotiation successful
0810 00:46:00.045781 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 00:46:00.060694 8237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:00.055552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:60033 (local address 127.28.6.1:44791)
0810 00:46:00.055734 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:00.055739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:00.056290 (+ 551us) server_negotiation.cc:407] Connection header received
0810 00:46:00.056419 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:00.056423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:00.056488 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:00.056570 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:46:00.056993 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:00.057676 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:00.059147 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:00.059634 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:00.059743 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:00.060133 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:46:00.060232 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:46:00.060299 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:00.060423 (+ 124us) server_negotiation.cc:299] Negotiation successful
0810 00:46:00.060483 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 00:46:00.060806 8313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:00.055475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:44791 (local address 127.28.6.3:60033)
0810 00:46:00.056187 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:00.056204 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:00.056316 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:00.056595 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:00.056598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:00.056607 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 00:46:00.056869 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:00.056876 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:00.057844 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:00.057847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:00.058989 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:00.059002 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:00.059803 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:00.059807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:00.059931 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:00.059987 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 00:46:00.060318 (+ 331us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 00:46:00.060327 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 00:46:00.060387 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 00:46:00.060476 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":363,"threads_started":1}
I20260810 00:46:00.069154 7694 raft_consensus.cc:1261] T 906f3feeda914f78b6028dd5e70ddb2d P e74119537bff4b4ab0ca81a2ecc85fc8 [term 1 FOLLOWER]: Refusing update from remote peer 5e4f9e7de3134a81a712da7e20a37f20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:46:00.069795 7879 raft_consensus.cc:1261] T 906f3feeda914f78b6028dd5e70ddb2d P 49ede907cb434995b3b28267e5a43baa [term 1 FOLLOWER]: Refusing update from remote peer 5e4f9e7de3134a81a712da7e20a37f20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:46:00.069926 8306 consensus_queue.cc:1035] T 906f3feeda914f78b6028dd5e70ddb2d P 5e4f9e7de3134a81a712da7e20a37f20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e74119537bff4b4ab0ca81a2ecc85fc8" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:46:00.070839 8306 consensus_queue.cc:1035] T 906f3feeda914f78b6028dd5e70ddb2d P 5e4f9e7de3134a81a712da7e20a37f20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49ede907cb434995b3b28267e5a43baa" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:46:00.095669 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:00.090161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:40347 (local address 127.28.6.3:41251)
0810 00:46:00.090795 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:00.090799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:00.090815 (+ 16us) server_negotiation.cc:407] Connection header received
0810 00:46:00.090938 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:00.090946 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:00.091048 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:00.091163 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:46:00.091576 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:00.092324 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:00.093762 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:00.094335 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:00.094416 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:00.095010 (+ 594us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:46:00.095154 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:46:00.095259 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:00.095395 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 00:46:00.095469 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553}
I20260810 00:46:00.095706 8318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:00.090093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:41251 (local address 127.28.6.1:40347)
0810 00:46:00.090661 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:00.090680 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:00.090790 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:00.091180 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:00.091184 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:00.091197 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 00:46:00.091448 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:00.091459 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:00.092455 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:00.092462 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:00.093561 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:00.093574 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:00.094585 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:00.094589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:00.094793 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:00.094830 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 00:46:00.095264 (+ 434us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 00:46:00.095274 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 00:46:00.095352 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 00:46:00.095448 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":257,"threads_started":1}
I20260810 00:46:00.128011 7462 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:46:00.128270 7462 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:46:00.131505 7462 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 89582f0c9b9943f2ba432c32687a6435: Sending DeleteTablet for 3 replicas of tablet 906f3feeda914f78b6028dd5e70ddb2d
I20260810 00:46:00.132409 7853 tablet_service.cc:1514] Processing DeleteTablet for tablet 906f3feeda914f78b6028dd5e70ddb2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:46:00 UTC) from {username='oryx', principal='oryx/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:53188
I20260810 00:46:00.132447 7657 tablet_service.cc:1514] Processing DeleteTablet for tablet 906f3feeda914f78b6028dd5e70ddb2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:46:00 UTC) from {username='oryx', principal='oryx/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:47078
I20260810 00:46:00.132568 8049 tablet_service.cc:1514] Processing DeleteTablet for tablet 906f3feeda914f78b6028dd5e70ddb2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:46:00 UTC) from {username='oryx', principal='oryx/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:60896
I20260810 00:46:00.132723 8262 tablet_replica.cc:330] T 906f3feeda914f78b6028dd5e70ddb2d P e74119537bff4b4ab0ca81a2ecc85fc8: stopping tablet replica
I20260810 00:46:00.132738 8261 tablet_replica.cc:330] T 906f3feeda914f78b6028dd5e70ddb2d P 49ede907cb434995b3b28267e5a43baa: stopping tablet replica
I20260810 00:46:00.132905 8261 raft_consensus.cc:2229] T 906f3feeda914f78b6028dd5e70ddb2d P 49ede907cb434995b3b28267e5a43baa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:46:00.132906 8262 raft_consensus.cc:2229] T 906f3feeda914f78b6028dd5e70ddb2d P e74119537bff4b4ab0ca81a2ecc85fc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:46:00.133105 8261 raft_consensus.cc:2258] T 906f3feeda914f78b6028dd5e70ddb2d P 49ede907cb434995b3b28267e5a43baa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:46:00.133105 8262 raft_consensus.cc:2258] T 906f3feeda914f78b6028dd5e70ddb2d P e74119537bff4b4ab0ca81a2ecc85fc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:46:00.133404 8260 tablet_replica.cc:330] T 906f3feeda914f78b6028dd5e70ddb2d P 5e4f9e7de3134a81a712da7e20a37f20: stopping tablet replica
I20260810 00:46:00.133596 8260 raft_consensus.cc:2229] T 906f3feeda914f78b6028dd5e70ddb2d P 5e4f9e7de3134a81a712da7e20a37f20 [term 1 LEADER]: Raft consensus shutting down.
I20260810 00:46:00.134135 8260 raft_consensus.cc:2258] T 906f3feeda914f78b6028dd5e70ddb2d P 5e4f9e7de3134a81a712da7e20a37f20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:46:00.134552 8262 ts_tablet_manager.cc:1905] T 906f3feeda914f78b6028dd5e70ddb2d P e74119537bff4b4ab0ca81a2ecc85fc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:46:00.134799 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 7568
I20260810 00:46:00.135660 8260 ts_tablet_manager.cc:1905] T 906f3feeda914f78b6028dd5e70ddb2d P 5e4f9e7de3134a81a712da7e20a37f20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:46:00.142204 8262 ts_tablet_manager.cc:1918] T 906f3feeda914f78b6028dd5e70ddb2d P e74119537bff4b4ab0ca81a2ecc85fc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 00:46:00.142380 8262 log.cc:1196] T 906f3feeda914f78b6028dd5e70ddb2d P e74119537bff4b4ab0ca81a2ecc85fc8: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-0/wal/wals/906f3feeda914f78b6028dd5e70ddb2d
I20260810 00:46:00.142834 8262 ts_tablet_manager.cc:1939] T 906f3feeda914f78b6028dd5e70ddb2d P e74119537bff4b4ab0ca81a2ecc85fc8: Deleting consensus metadata
I20260810 00:46:00.145896 7441 catalog_manager.cc:4841] TS e74119537bff4b4ab0ca81a2ecc85fc8 (127.28.6.1:44791): tablet 906f3feeda914f78b6028dd5e70ddb2d (table test-table [id=9ec6023eed644a5eb3c2c81e4b95246a]) successfully deleted
I20260810 00:46:00.146106 8260 ts_tablet_manager.cc:1918] T 906f3feeda914f78b6028dd5e70ddb2d P 5e4f9e7de3134a81a712da7e20a37f20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 00:46:00.146294 8260 log.cc:1196] T 906f3feeda914f78b6028dd5e70ddb2d P 5e4f9e7de3134a81a712da7e20a37f20: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-2/wal/wals/906f3feeda914f78b6028dd5e70ddb2d
I20260810 00:46:00.146920 8260 ts_tablet_manager.cc:1939] T 906f3feeda914f78b6028dd5e70ddb2d P 5e4f9e7de3134a81a712da7e20a37f20: Deleting consensus metadata
I20260810 00:46:00.147871 8261 ts_tablet_manager.cc:1905] T 906f3feeda914f78b6028dd5e70ddb2d P 49ede907cb434995b3b28267e5a43baa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:46:00.153208 8261 ts_tablet_manager.cc:1918] T 906f3feeda914f78b6028dd5e70ddb2d P 49ede907cb434995b3b28267e5a43baa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 00:46:00.153363 8261 log.cc:1196] T 906f3feeda914f78b6028dd5e70ddb2d P 49ede907cb434995b3b28267e5a43baa: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786322687226232-28696-0/minicluster-data/ts-1/wal/wals/906f3feeda914f78b6028dd5e70ddb2d
I20260810 00:46:00.153841 8261 ts_tablet_manager.cc:1939] T 906f3feeda914f78b6028dd5e70ddb2d P 49ede907cb434995b3b28267e5a43baa: Deleting consensus metadata
I20260810 00:46:00.155033 7442 catalog_manager.cc:4841] TS 49ede907cb434995b3b28267e5a43baa (127.28.6.2:35007): tablet 906f3feeda914f78b6028dd5e70ddb2d (table test-table [id=9ec6023eed644a5eb3c2c81e4b95246a]) successfully deleted
I20260810 00:46:00.155845 7441 catalog_manager.cc:4841] TS 5e4f9e7de3134a81a712da7e20a37f20 (127.28.6.3:41251): tablet 906f3feeda914f78b6028dd5e70ddb2d (table test-table [id=9ec6023eed644a5eb3c2c81e4b95246a]) successfully deleted
Aug 10 00:46:00 dist-test-slave-1nl6 krb5kdc[7401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.6.1@KRBTEST.COM
I20260810 00:46:00.164927 8237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:00.136615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47108 (local address 127.28.6.1:44791)
0810 00:46:00.136787 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:00.136794 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:00.147795 (+ 11001us) server_negotiation.cc:407] Connection header received
0810 00:46:00.147980 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:00.147988 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:00.148065 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:00.148185 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:00.149705 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:00.150780 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:00.152618 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:00.153303 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:00.153455 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:00.158214 (+ 4759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:46:00.158232 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:46:00.158253 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:46:00.158292 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:46:00.162103 (+ 3811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:46:00.162743 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:46:00.162753 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:46:00.162757 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:46:00.162975 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:46:00.163351 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:46:00.163356 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:46:00.163359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:46:00.163790 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:46:00.164130 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:00.164308 (+ 178us) server_negotiation.cc:299] Negotiation successful
0810 00:46:00.164542 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 00:46:00.165656 7751 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:46:00.314699 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 7568
I20260810 00:46:00.359692 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 7762
Aug 10 00:46:00 dist-test-slave-1nl6 krb5kdc[7401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.6.2@KRBTEST.COM
I20260810 00:46:00.376485 8303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:00.360508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53202 (local address 127.28.6.2:35007)
0810 00:46:00.360687 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:00.360692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:00.360767 (+ 75us) server_negotiation.cc:407] Connection header received
0810 00:46:00.360975 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:00.360979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:00.361060 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:00.361141 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:00.362777 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:00.363503 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:00.365386 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:00.365791 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:00.365877 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:00.370728 (+ 4851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:46:00.370741 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:46:00.370755 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:46:00.370786 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:46:00.373696 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:46:00.374365 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:46:00.374374 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:46:00.374376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:46:00.374507 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:46:00.374929 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:46:00.374935 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:46:00.374940 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:46:00.375294 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:46:00.375607 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:00.375908 (+ 301us) server_negotiation.cc:299] Negotiation successful
0810 00:46:00.376159 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 00:46:00.377126 7940 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:46:00.476516 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 7762
I20260810 00:46:00.512985 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 7956
Aug 10 00:46:00 dist-test-slave-1nl6 krb5kdc[7401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.6.3@KRBTEST.COM
I20260810 00:46:00.529551 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:00.513853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60922 (local address 127.28.6.3:41251)
0810 00:46:00.514112 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:00.514118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:00.514139 (+ 21us) server_negotiation.cc:407] Connection header received
0810 00:46:00.514271 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:00.514277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:00.514358 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:00.514499 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:00.515917 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:00.516759 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:00.518048 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:00.518532 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:00.518670 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:00.523125 (+ 4455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:46:00.523141 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:46:00.523160 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:46:00.523195 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:46:00.526471 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:46:00.527251 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:46:00.527267 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:46:00.527271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:46:00.527410 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:46:00.527821 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:46:00.527826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:46:00.527830 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:46:00.528301 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:46:00.528626 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:00.529006 (+ 380us) server_negotiation.cc:299] Negotiation successful
0810 00:46:00.529252 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 00:46:00.530565 8144 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:46:00.668172 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 7956
I20260810 00:46:00.701665 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 7423
I20260810 00:46:00.703013 7522 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:46:00.819300 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 7423
2026-08-10T00:46:00Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6236 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:46:00 dist-test-slave-1nl6 krb5kdc[8331](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:46:00 dist-test-slave-1nl6 krb5kdc[8331](info): set up 2 sockets
Aug 10 00:46:00 dist-test-slave-1nl6 krb5kdc[8331](info): commencing operation
krb5kdc: starting...
W20260810 00:46:02.115852 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.244s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:46:02 dist-test-slave-1nl6 krb5kdc[8331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:46:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:46:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.6.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
I20260810 00:46:02.343973 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:41159
--webserver_interface=127.28.6.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:42689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:41159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:46:02.670197 8354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:46:02.670464 8354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:46:02.670639 8354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:46:02.677569 8354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:46:02.677690 8354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:46:02.677775 8354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:46:02.677874 8354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:46:02.677963 8354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:46:02.687252 8354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:42689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:41159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:41159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:46:02.688473 8354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:46:02.689814 8354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:46:02.701669 8359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:02.701630 8362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:02.701706 8360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:46:02.702826 8354 server_base.cc:1034] running on GCE node
I20260810 00:46:02.703606 8354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:46:02.704933 8354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:46:02.706131 8354 hybrid_clock.cc:648] HybridClock initialized: now 1786322762706079 us; error 50 us; skew 500 ppm
I20260810 00:46:02.706487 8354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-10T00:46:02Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1952 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:46:02 dist-test-slave-1nl6 krb5kdc[8372](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:46:02 dist-test-slave-1nl6 krb5kdc[8372](info): set up 2 sockets
Aug 10 00:46:02 dist-test-slave-1nl6 krb5kdc[8372](info): commencing operation
krb5kdc: starting...
W20260810 00:46:04.059935 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.235s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:46:04 dist-test-slave-1nl6 krb5kdc[8372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:46:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:46:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.6.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
I20260810 00:46:04.294248 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:46485
--webserver_interface=127.28.6.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:43967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:46485
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:46:04.624761 8391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:46:04.625041 8391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:46:04.625214 8391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:46:04.632472 8391 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:46:04.632575 8391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:46:04.632680 8391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:46:04.632795 8391 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:46:04.632879 8391 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:46:04.642547 8391 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:43967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:46485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:46485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:46:04.643890 8391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:46:04.645237 8391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:46:04.657086 8398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:04.657176 8397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:04.657136 8400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:46:04.658016 8391 server_base.cc:1034] running on GCE node
I20260810 00:46:04.658861 8391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:46:04.660356 8391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:46:04.661572 8391 hybrid_clock.cc:648] HybridClock initialized: now 1786322764661525 us; error 41 us; skew 500 ppm
I20260810 00:46:04.661927 8391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:46:04 dist-test-slave-1nl6 krb5kdc[8372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322764, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:46:04.668687 8391 init.cc:377] Logged in from keytab as kudu/127.28.6.62@KRBTEST.COM (short username kudu)
I20260810 00:46:04.670219 8391 webserver.cc:460] Webserver started at http://127.28.6.62:44583/ using document root <none> and password file <none>
I20260810 00:46:04.671000 8391 fs_manager.cc:359] Metadata directory not provided
I20260810 00:46:04.671140 8391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:46:04.671482 8391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:46:04.674446 8391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "93a2698326444a5891609266e0620d2d"
format_stamp: "Formatted at 2026-08-10 00:46:04 on dist-test-slave-1nl6"
server_key: "501e63efde314328fc100f6bf808ce71"
server_key_iv: "09b3708cd79da2770f71a611c83803ba"
server_key_version: "encryptionkey@0"
I20260810 00:46:04.675247 8391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "93a2698326444a5891609266e0620d2d"
format_stamp: "Formatted at 2026-08-10 00:46:04 on dist-test-slave-1nl6"
server_key: "501e63efde314328fc100f6bf808ce71"
server_key_iv: "09b3708cd79da2770f71a611c83803ba"
server_key_version: "encryptionkey@0"
I20260810 00:46:04.680428 8391 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 00:46:04.683759 8410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:04.685040 8391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 00:46:04.685230 8391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "93a2698326444a5891609266e0620d2d"
format_stamp: "Formatted at 2026-08-10 00:46:04 on dist-test-slave-1nl6"
server_key: "501e63efde314328fc100f6bf808ce71"
server_key_iv: "09b3708cd79da2770f71a611c83803ba"
server_key_version: "encryptionkey@0"
I20260810 00:46:04.685424 8391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:46:04.718535 8391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:46:04.722476 8391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:46:04.722774 8391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:46:04.741521 8391 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:46485
I20260810 00:46:04.741565 8483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:46485 every 8 connection(s)
I20260810 00:46:04.742867 8391 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:46:04.745570 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 8391
I20260810 00:46:04.745908 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:46:04.746783 28696 external_mini_cluster.cc:1428] Setting key 7a3449c5f41b6902d63a2541d222e45b
I20260810 00:46:04.747445 8486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:46:04.762076 8486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d: Bootstrap starting.
Aug 10 00:46:04 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:46:04.765255 8486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d: Neither blocks nor log segments found. Creating new log.
I20260810 00:46:04.766322 8486 log.cc:824] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d: Log is configured to *not* fsync() on all Append() calls
I20260810 00:46:04.769037 8486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d: No bootstrap required, opened a new log
I20260810 00:46:04.770547 8489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:04.748911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53616 (local address 127.28.6.62:46485)
0810 00:46:04.749779 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:04.749791 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:04.749833 (+ 42us) server_negotiation.cc:407] Connection header received
0810 00:46:04.752191 (+ 2358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:04.752214 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:04.752606 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:04.752924 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:04.754809 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:04.756111 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:04.757972 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:04.758651 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:04.758784 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:04.763547 (+ 4763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:46:04.763556 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:46:04.763569 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:46:04.763600 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:46:04.766669 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:46:04.767456 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:46:04.767463 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:46:04.767469 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:46:04.767648 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:46:04.768036 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:46:04.768042 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:46:04.768047 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:46:04.768531 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:46:04.768835 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:04.769283 (+ 448us) server_negotiation.cc:299] Negotiation successful
0810 00:46:04.769663 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":408,"threads_started":1}
I20260810 00:46:04.777133 8486 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a2698326444a5891609266e0620d2d" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 46485 } }
I20260810 00:46:04.777522 8486 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:46:04.777653 8486 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93a2698326444a5891609266e0620d2d, State: Initialized, Role: FOLLOWER
I20260810 00:46:04.778131 8486 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a2698326444a5891609266e0620d2d" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 46485 } }
I20260810 00:46:04.778337 8486 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:46:04.778434 8486 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:46:04.778558 8486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:46:04.780910 8486 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a2698326444a5891609266e0620d2d" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 46485 } }
I20260810 00:46:04.781384 8486 leader_election.cc:302] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93a2698326444a5891609266e0620d2d; no voters:
I20260810 00:46:04.781947 8486 leader_election.cc:288] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:46:04.782073 8495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:46:04.782953 8495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d [term 1 LEADER]: Becoming Leader. State: Replica: 93a2698326444a5891609266e0620d2d, State: Running, Role: LEADER
I20260810 00:46:04.783480 8495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a2698326444a5891609266e0620d2d" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 46485 } }
I20260810 00:46:04.784260 8486 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:46:04.786339 8499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93a2698326444a5891609266e0620d2d. Latest consensus state: current_term: 1 leader_uuid: "93a2698326444a5891609266e0620d2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a2698326444a5891609266e0620d2d" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 46485 } } }
I20260810 00:46:04.786367 8497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93a2698326444a5891609266e0620d2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a2698326444a5891609266e0620d2d" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 46485 } } }
I20260810 00:46:04.786878 8499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d [sys.catalog]: This master's current role is: LEADER
I20260810 00:46:04.787027 8497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d [sys.catalog]: This master's current role is: LEADER
I20260810 00:46:04.789789 8504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:46:04.795070 8504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:46:04.804685 8504 catalog_manager.cc:1347] Generated new cluster ID: 3eae2a092cfb4684a8da771dbe1c209e
I20260810 00:46:04.804811 8504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:46:04.829222 8504 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:46:04.830472 8504 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:46:04.838084 8504 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d: Generated new TSK 0
I20260810 00:46:04.838817 8504 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.6.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab.
Entry for principal kudu/127.28.6.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab.
I20260810 00:46:04.912386 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.1:0
--local_ip_for_outbound_sockets=127.28.6.1
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:46485
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:43967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:46:05.244472 8526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:46:05.244741 8526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:46:05.244835 8526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:46:05.251639 8526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:46:05.251736 8526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:46:05.251842 8526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.1
I20260810 00:46:05.260800 8526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:43967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:46485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.1
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:46:05.262042 8526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:46:05.263309 8526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:46:05.276189 8533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:05.276149 8536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:05.276223 8534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:46:05.277542 8526 server_base.cc:1034] running on GCE node
I20260810 00:46:05.278146 8526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:46:05.279100 8526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:46:05.280328 8526 hybrid_clock.cc:648] HybridClock initialized: now 1786322765280290 us; error 46 us; skew 500 ppm
I20260810 00:46:05.280653 8526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:46:05 dist-test-slave-1nl6 krb5kdc[8372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322765, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:46:05.283723 8526 init.cc:377] Logged in from keytab as kudu/127.28.6.1@KRBTEST.COM (short username kudu)
I20260810 00:46:05.285162 8526 webserver.cc:460] Webserver started at http://127.28.6.1:41039/ using document root <none> and password file <none>
I20260810 00:46:05.285944 8526 fs_manager.cc:359] Metadata directory not provided
I20260810 00:46:05.286090 8526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:46:05.286406 8526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:46:05.289326 8526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-0/data/instance:
uuid: "80073accca294a6ba4a1733482669960"
format_stamp: "Formatted at 2026-08-10 00:46:05 on dist-test-slave-1nl6"
server_key: "f7ed879b337721187e18648cb83e411e"
server_key_iv: "6d9af6acc7fa496186ce491ac5416fba"
server_key_version: "encryptionkey@0"
I20260810 00:46:05.290269 8526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance:
uuid: "80073accca294a6ba4a1733482669960"
format_stamp: "Formatted at 2026-08-10 00:46:05 on dist-test-slave-1nl6"
server_key: "f7ed879b337721187e18648cb83e411e"
server_key_iv: "6d9af6acc7fa496186ce491ac5416fba"
server_key_version: "encryptionkey@0"
I20260810 00:46:05.295324 8526 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260810 00:46:05.298584 8546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:05.299861 8526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 00:46:05.300045 8526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-0/wal
uuid: "80073accca294a6ba4a1733482669960"
format_stamp: "Formatted at 2026-08-10 00:46:05 on dist-test-slave-1nl6"
server_key: "f7ed879b337721187e18648cb83e411e"
server_key_iv: "6d9af6acc7fa496186ce491ac5416fba"
server_key_version: "encryptionkey@0"
I20260810 00:46:05.300236 8526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:46:05.318352 8526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:46:05.322270 8526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:46:05.322566 8526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:46:05.323523 8526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:46:05.324856 8526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:46:05.324962 8526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:05.325121 8526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:46:05.325198 8526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:05.358259 8526 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.1:36441
I20260810 00:46:05.358310 8709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.1:36441 every 8 connection(s)
I20260810 00:46:05.359596 8526 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
I20260810 00:46:05.367305 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 8526
I20260810 00:46:05.367506 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance
I20260810 00:46:05.368119 28696 external_mini_cluster.cc:1428] Setting key ddc7adb1195d0b3254324ea692146b34
Aug 10 00:46:05 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322765, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:46:05.379339 8714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:05.362103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:56523 (local address 127.28.6.62:46485)
0810 00:46:05.362570 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:05.362574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:05.363436 (+ 862us) server_negotiation.cc:407] Connection header received
0810 00:46:05.365199 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:05.365205 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:05.365302 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:05.365426 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:05.366857 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:05.367739 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:05.369402 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:05.369895 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:05.370027 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:05.373602 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:46:05.373607 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:46:05.373610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:46:05.373641 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:46:05.376300 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:46:05.376996 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:46:05.377002 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:46:05.377007 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:46:05.377127 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:46:05.377560 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:46:05.377565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:46:05.377568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:46:05.377828 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:46:05.377987 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:05.378657 (+ 670us) server_negotiation.cc:299] Negotiation successful
0810 00:46:05.378994 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":259,"threads_started":1}
I20260810 00:46:05.380234 8715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:05.362486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.1:56523)
0810 00:46:05.363281 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:05.363319 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:05.364900 (+ 1581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:05.365648 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:05.365656 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:05.366151 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:05.366672 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:05.366689 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:05.367890 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:05.367895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:05.369265 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:05.369277 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:05.370119 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:05.370127 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:05.370365 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:05.371048 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:05.371075 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:05.373397 (+ 2322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:46:05.376443 (+ 3046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:46:05.376452 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:46:05.376467 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:46:05.376867 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:46:05.377246 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:46:05.377251 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:46:05.377255 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:46:05.377443 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:46:05.377966 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:46:05.377973 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:46:05.378305 (+ 332us) client_negotiation.cc:769] Sending connection context
0810 00:46:05.378553 (+ 248us) client_negotiation.cc:240] Negotiation successful
0810 00:46:05.379075 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":375,"threads_started":1}
I20260810 00:46:05.381737 8710 heartbeater.cc:347] Connected to a master server at 127.28.6.62:46485
I20260810 00:46:05.382056 8710 heartbeater.cc:464] Registering TS with master...
I20260810 00:46:05.382887 8710 heartbeater.cc:511] Master 127.28.6.62:46485 requested a full tablet report, sending...
I20260810 00:46:05.385483 8431 ts_manager.cc:194] Registered new tserver with Master: 80073accca294a6ba4a1733482669960 (127.28.6.1:36441)
I20260810 00:46:05.387698 8431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.1@KRBTEST.COM'} at 127.28.6.1:56523
WARNING: no policy specified for kudu/127.28.6.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab.
Entry for principal kudu/127.28.6.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab.
I20260810 00:46:05.424083 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.2:0
--local_ip_for_outbound_sockets=127.28.6.2
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:46485
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:43967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:46:05.758706 8719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:46:05.759022 8719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:46:05.759192 8719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:46:05.766110 8719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:46:05.766207 8719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:46:05.766443 8719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.2
I20260810 00:46:05.775719 8719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:43967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:46485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.2
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:46:05.776847 8719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:46:05.778083 8719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:46:05.791180 8730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:05.791193 8727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:05.791231 8728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:46:05.792439 8719 server_base.cc:1034] running on GCE node
I20260810 00:46:05.793005 8719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:46:05.793879 8719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:46:05.795100 8719 hybrid_clock.cc:648] HybridClock initialized: now 1786322765795054 us; error 65 us; skew 500 ppm
I20260810 00:46:05.795419 8719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:46:05 dist-test-slave-1nl6 krb5kdc[8372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322765, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:46:05.798663 8719 init.cc:377] Logged in from keytab as kudu/127.28.6.2@KRBTEST.COM (short username kudu)
I20260810 00:46:05.800138 8719 webserver.cc:460] Webserver started at http://127.28.6.2:46395/ using document root <none> and password file <none>
I20260810 00:46:05.800806 8719 fs_manager.cc:359] Metadata directory not provided
I20260810 00:46:05.800956 8719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:46:05.801290 8719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:46:05.804526 8719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-1/data/instance:
uuid: "74b64516dd9549be9b73b7ca0d872a97"
format_stamp: "Formatted at 2026-08-10 00:46:05 on dist-test-slave-1nl6"
server_key: "ff5d2057199a1b208eaf0cfa8b589f09"
server_key_iv: "75a808417bafbdec32e89b866c966750"
server_key_version: "encryptionkey@0"
I20260810 00:46:05.805379 8719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance:
uuid: "74b64516dd9549be9b73b7ca0d872a97"
format_stamp: "Formatted at 2026-08-10 00:46:05 on dist-test-slave-1nl6"
server_key: "ff5d2057199a1b208eaf0cfa8b589f09"
server_key_iv: "75a808417bafbdec32e89b866c966750"
server_key_version: "encryptionkey@0"
I20260810 00:46:05.810371 8719 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 00:46:05.813587 8739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:05.814814 8719 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 00:46:05.815007 8719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-1/wal
uuid: "74b64516dd9549be9b73b7ca0d872a97"
format_stamp: "Formatted at 2026-08-10 00:46:05 on dist-test-slave-1nl6"
server_key: "ff5d2057199a1b208eaf0cfa8b589f09"
server_key_iv: "75a808417bafbdec32e89b866c966750"
server_key_version: "encryptionkey@0"
I20260810 00:46:05.815214 8719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:46:05.828790 8719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:46:05.832926 8719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:46:05.833230 8719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:46:05.834167 8719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:46:05.835500 8719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:46:05.835606 8719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:05.835764 8719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:46:05.835843 8719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:05.869653 8719 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.2:33225
I20260810 00:46:05.869709 8900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.2:33225 every 8 connection(s)
I20260810 00:46:05.871304 8719 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
I20260810 00:46:05.875110 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 8719
I20260810 00:46:05.875306 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance
I20260810 00:46:05.875794 28696 external_mini_cluster.cc:1428] Setting key d5770a7d33b0310aa48526d0a172b523
Aug 10 00:46:05 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322765, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:46:05.891417 8714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:05.873880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:51211 (local address 127.28.6.62:46485)
0810 00:46:05.874057 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:05.874062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:05.875249 (+ 1187us) server_negotiation.cc:407] Connection header received
0810 00:46:05.877366 (+ 2117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:05.877372 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:05.877466 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:05.877578 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:05.879058 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:05.879755 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:05.881247 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:05.881702 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:05.881853 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:05.885420 (+ 3567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:46:05.885425 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:46:05.885428 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:46:05.885458 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:46:05.887978 (+ 2520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:46:05.888870 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:46:05.888874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:46:05.888877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:46:05.889010 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:46:05.889515 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:46:05.889519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:46:05.889521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:46:05.889850 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:46:05.890036 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:05.890889 (+ 853us) server_negotiation.cc:299] Negotiation successful
0810 00:46:05.891182 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 00:46:05.892864 8904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:05.874339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.2:51211)
0810 00:46:05.875094 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:05.875131 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:05.877107 (+ 1976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:05.877811 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:05.877820 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:05.878312 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:05.878851 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:05.878868 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:05.879928 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:05.879933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:05.881081 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:05.881093 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:05.881927 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:05.881934 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:05.882180 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:05.882857 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:05.882879 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:05.885165 (+ 2286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:46:05.888133 (+ 2968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:46:05.888142 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:46:05.888155 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:46:05.888737 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:46:05.889144 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:46:05.889152 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:46:05.889158 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:46:05.889402 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:46:05.889990 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:46:05.890000 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:46:05.890431 (+ 431us) client_negotiation.cc:769] Sending connection context
0810 00:46:05.890718 (+ 287us) client_negotiation.cc:240] Negotiation successful
0810 00:46:05.891397 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":297,"threads_started":1}
I20260810 00:46:05.894374 8902 heartbeater.cc:347] Connected to a master server at 127.28.6.62:46485
I20260810 00:46:05.894685 8902 heartbeater.cc:464] Registering TS with master...
I20260810 00:46:05.895471 8902 heartbeater.cc:511] Master 127.28.6.62:46485 requested a full tablet report, sending...
I20260810 00:46:05.896927 8431 ts_manager.cc:194] Registered new tserver with Master: 74b64516dd9549be9b73b7ca0d872a97 (127.28.6.2:33225)
I20260810 00:46:05.897709 8431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.2@KRBTEST.COM'} at 127.28.6.2:51211
WARNING: no policy specified for kudu/127.28.6.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab.
Entry for principal kudu/127.28.6.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab.
I20260810 00:46:05.929821 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.3:0
--local_ip_for_outbound_sockets=127.28.6.3
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:46485
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:43967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:46:06.262795 8908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:46:06.263124 8908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:46:06.263303 8908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:46:06.270169 8908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:46:06.270331 8908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:46:06.270498 8908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.3
I20260810 00:46:06.279745 8908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:43967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:46485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.3
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:46:06.280889 8908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:46:06.282115 8908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:46:06.295007 8917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:06.295061 8914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:06.295164 8913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:46:06.295993 8908 server_base.cc:1034] running on GCE node
I20260810 00:46:06.296628 8908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:46:06.297520 8908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:46:06.298758 8908 hybrid_clock.cc:648] HybridClock initialized: now 1786322766298705 us; error 58 us; skew 500 ppm
I20260810 00:46:06.299114 8908 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:46:06 dist-test-slave-1nl6 krb5kdc[8372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322766, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:46:06.302248 8908 init.cc:377] Logged in from keytab as kudu/127.28.6.3@KRBTEST.COM (short username kudu)
I20260810 00:46:06.303683 8908 webserver.cc:460] Webserver started at http://127.28.6.3:37017/ using document root <none> and password file <none>
I20260810 00:46:06.304266 8908 fs_manager.cc:359] Metadata directory not provided
I20260810 00:46:06.304423 8908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:46:06.304699 8908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:46:06.307554 8908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-2/data/instance:
uuid: "7084d42358aa41bc9690ddf2b2889b80"
format_stamp: "Formatted at 2026-08-10 00:46:06 on dist-test-slave-1nl6"
server_key: "c683a9a2c0ef4c867353334664f02a0f"
server_key_iv: "997d85a52a6d0a74dec5dc4fb8ff05f3"
server_key_version: "encryptionkey@0"
I20260810 00:46:06.308315 8908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance:
uuid: "7084d42358aa41bc9690ddf2b2889b80"
format_stamp: "Formatted at 2026-08-10 00:46:06 on dist-test-slave-1nl6"
server_key: "c683a9a2c0ef4c867353334664f02a0f"
server_key_iv: "997d85a52a6d0a74dec5dc4fb8ff05f3"
server_key_version: "encryptionkey@0"
I20260810 00:46:06.313421 8908 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 00:46:06.316747 8927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:06.317952 8908 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 00:46:06.318125 8908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-2/wal
uuid: "7084d42358aa41bc9690ddf2b2889b80"
format_stamp: "Formatted at 2026-08-10 00:46:06 on dist-test-slave-1nl6"
server_key: "c683a9a2c0ef4c867353334664f02a0f"
server_key_iv: "997d85a52a6d0a74dec5dc4fb8ff05f3"
server_key_version: "encryptionkey@0"
I20260810 00:46:06.318280 8908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:46:06.340180 8908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:46:06.343931 8908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:46:06.344167 8908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:46:06.345023 8908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:46:06.346277 8908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:46:06.346364 8908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:06.346447 8908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:46:06.346493 8908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:06.380631 8908 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.3:41379
I20260810 00:46:06.380717 9099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.3:41379 every 8 connection(s)
I20260810 00:46:06.381896 8908 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
I20260810 00:46:06.390872 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 8908
I20260810 00:46:06.391104 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance
I20260810 00:46:06.391422 8710 heartbeater.cc:503] Master 127.28.6.62:46485 was elected leader, sending a full tablet report...
I20260810 00:46:06.391621 28696 external_mini_cluster.cc:1428] Setting key eca98388eac566ac5979196c4eda0025
Aug 10 00:46:06 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322766, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:46:06.402467 8714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:06.384589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:43559 (local address 127.28.6.62:46485)
0810 00:46:06.384769 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:06.384773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:06.386001 (+ 1228us) server_negotiation.cc:407] Connection header received
0810 00:46:06.387930 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:06.387937 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:06.388034 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:06.388163 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:06.389775 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:06.390626 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:06.393045 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:06.393567 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:06.393683 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:06.397224 (+ 3541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:46:06.397228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:46:06.397232 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:46:06.397261 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:46:06.399788 (+ 2527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:46:06.400498 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:46:06.400502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:46:06.400504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:46:06.400629 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:46:06.401074 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:46:06.401077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:46:06.401079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:46:06.401319 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:46:06.401435 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:06.402067 (+ 632us) server_negotiation.cc:299] Negotiation successful
0810 00:46:06.402224 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 00:46:06.403474 9106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:06.385012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.3:43559)
0810 00:46:06.385848 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:06.385885 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:06.387651 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:06.388414 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:06.388421 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:06.388956 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:06.389593 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:06.389624 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:06.390814 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:06.390818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:06.392804 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:06.392818 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:06.393849 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:06.393858 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:06.394133 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:06.394818 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:06.394840 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:06.397047 (+ 2207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:46:06.399923 (+ 2876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:46:06.399929 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:46:06.399939 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:46:06.400381 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:46:06.400722 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:46:06.400725 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:46:06.400727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:46:06.400963 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:46:06.401432 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:46:06.401438 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:46:06.401725 (+ 287us) client_negotiation.cc:769] Sending connection context
0810 00:46:06.401945 (+ 220us) client_negotiation.cc:240] Negotiation successful
0810 00:46:06.402311 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":324,"threads_started":1}
I20260810 00:46:06.404933 9100 heartbeater.cc:347] Connected to a master server at 127.28.6.62:46485
I20260810 00:46:06.405189 9100 heartbeater.cc:464] Registering TS with master...
I20260810 00:46:06.405809 9100 heartbeater.cc:511] Master 127.28.6.62:46485 requested a full tablet report, sending...
I20260810 00:46:06.407276 8431 ts_manager.cc:194] Registered new tserver with Master: 7084d42358aa41bc9690ddf2b2889b80 (127.28.6.3:41379)
I20260810 00:46:06.408056 8431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.3@KRBTEST.COM'} at 127.28.6.3:43559
I20260810 00:46:06.419658 28696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.6.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.6.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.6.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.1.keytab.
Entry for principal oryx/127.28.6.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.1.keytab.
WARNING: no policy specified for oryx/127.28.6.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.6.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.6.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.2.keytab.
Entry for principal oryx/127.28.6.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.2.keytab.
WARNING: no policy specified for oryx/127.28.6.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.6.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.6.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.3.keytab.
Entry for principal oryx/127.28.6.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.3.keytab.
I20260810 00:46:06.554802 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 8526
Aug 10 00:46:06 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.1@KRBTEST.COM
I20260810 00:46:06.571183 9119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:06.556095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34462 (local address 127.28.6.1:36441)
0810 00:46:06.556744 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:06.556752 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:06.556780 (+ 28us) server_negotiation.cc:407] Connection header received
0810 00:46:06.556845 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:06.556852 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:06.557086 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:06.557254 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:06.558579 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:06.559706 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:06.560883 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:06.561452 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:06.561550 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:06.565470 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:46:06.565481 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:46:06.565496 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:46:06.565533 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:46:06.568618 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:46:06.569263 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:46:06.569270 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:46:06.569272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:46:06.569399 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:46:06.569761 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:46:06.569764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:46:06.569766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:46:06.570090 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:46:06.570324 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:06.570551 (+ 227us) server_negotiation.cc:299] Negotiation successful
0810 00:46:06.570804 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":313,"threads_started":1}
I20260810 00:46:06.572499 8705 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:46:06.656219 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 8526
I20260810 00:46:06.677891 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 8719
Aug 10 00:46:06 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.2@KRBTEST.COM
I20260810 00:46:06.694602 9121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:06.678856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42822 (local address 127.28.6.2:33225)
0810 00:46:06.679490 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:06.679497 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:06.679524 (+ 27us) server_negotiation.cc:407] Connection header received
0810 00:46:06.679587 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:06.679593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:06.679813 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:06.679946 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:06.681132 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:06.682147 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:06.683438 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:06.684061 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:06.684209 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:06.688517 (+ 4308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:46:06.688528 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:46:06.688541 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:46:06.688574 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:46:06.691781 (+ 3207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:46:06.692488 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:46:06.692496 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:46:06.692499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:46:06.692622 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:46:06.693014 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:46:06.693021 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:46:06.693029 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:46:06.693385 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:46:06.693727 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:06.693993 (+ 266us) server_negotiation.cc:299] Negotiation successful
0810 00:46:06.694261 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":315,"threads_started":1}
I20260810 00:46:06.695951 8896 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:46:06.787922 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 8719
I20260810 00:46:06.814303 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 8908
Aug 10 00:46:06 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.3@KRBTEST.COM
I20260810 00:46:06.831535 9124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:06.815478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47390 (local address 127.28.6.3:41379)
0810 00:46:06.816228 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:06.816241 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:06.816285 (+ 44us) server_negotiation.cc:407] Connection header received
0810 00:46:06.816355 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:06.816362 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:06.816573 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:06.816698 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:06.817994 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:06.819104 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:06.820390 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:06.820929 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:06.821029 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:06.825803 (+ 4774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:46:06.825814 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:46:06.825851 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:46:06.825897 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:46:06.828856 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:46:06.829490 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:46:06.829497 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:46:06.829499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:46:06.829680 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:46:06.830068 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:46:06.830071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:46:06.830073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:46:06.830410 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:46:06.830637 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:06.830933 (+ 296us) server_negotiation.cc:299] Negotiation successful
0810 00:46:06.831179 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":333,"threads_started":1}
I20260810 00:46:06.832826 9093 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:46:06.915874 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 8908
I20260810 00:46:06.942013 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 8391
I20260810 00:46:06.943306 8479 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:46:07.047614 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 8391
I20260810 00:46:07.079281 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:46485
--webserver_interface=127.28.6.62
--webserver_port=44583
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:43967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:46485
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:46:07.413379 9131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:46:07.413697 9131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:46:07.413874 9131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:46:07.420964 9131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:46:07.421105 9131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:46:07.421218 9131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:46:07.421326 9131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:46:07.421418 9131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:46:07.430840 9131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:43967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:46485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:46485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=44583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:46:07.432247 9131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:46:07.433645 9131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:46:07.445266 9137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:07.445348 9138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:07.445289 9140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:46:07.446746 9131 server_base.cc:1034] running on GCE node
I20260810 00:46:07.447500 9131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:46:07.448729 9131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:46:07.449963 9131 hybrid_clock.cc:648] HybridClock initialized: now 1786322767449908 us; error 42 us; skew 500 ppm
I20260810 00:46:07.450294 9131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:46:07 dist-test-slave-1nl6 krb5kdc[8372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322767, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:46:07.453346 9131 init.cc:377] Logged in from keytab as kudu/127.28.6.62@KRBTEST.COM (short username kudu)
I20260810 00:46:07.454766 9131 webserver.cc:460] Webserver started at http://127.28.6.62:44583/ using document root <none> and password file <none>
I20260810 00:46:07.455493 9131 fs_manager.cc:359] Metadata directory not provided
I20260810 00:46:07.455669 9131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:46:07.460791 9131 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 00:46:07.463490 9157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:07.464735 9131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 00:46:07.464910 9131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "93a2698326444a5891609266e0620d2d"
format_stamp: "Formatted at 2026-08-10 00:46:04 on dist-test-slave-1nl6"
server_key: "501e63efde314328fc100f6bf808ce71"
server_key_iv: "09b3708cd79da2770f71a611c83803ba"
server_key_version: "encryptionkey@0"
I20260810 00:46:07.465881 9131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:46:07.490386 9131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:46:07.494237 9131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:46:07.494483 9131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:46:07.514389 9131 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:46485
I20260810 00:46:07.514432 9229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:46485 every 8 connection(s)
I20260810 00:46:07.515969 9131 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:46:07.522787 9230 sys_catalog.cc:263] Verifying existing consensus state
I20260810 00:46:07.524374 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 9131
I20260810 00:46:07.525372 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.1:36441
--local_ip_for_outbound_sockets=127.28.6.1
--tserver_master_addrs=127.28.6.62:46485
--webserver_port=41039
--webserver_interface=127.28.6.1
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:43967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.6.1
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.1.keytab with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 00:46:07.526535 9230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d: Bootstrap starting.
I20260810 00:46:07.548862 9230 log.cc:824] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d: Log is configured to *not* fsync() on all Append() calls
I20260810 00:46:07.557863 9230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 00:46:07.558446 9230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d: Bootstrap complete.
I20260810 00:46:07.567408 9230 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a2698326444a5891609266e0620d2d" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 46485 } }
I20260810 00:46:07.568256 9230 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93a2698326444a5891609266e0620d2d, State: Initialized, Role: FOLLOWER
I20260810 00:46:07.568889 9230 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a2698326444a5891609266e0620d2d" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 46485 } }
I20260810 00:46:07.569140 9230 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:46:07.569305 9230 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:46:07.569504 9230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d [term 1 FOLLOWER]: Advancing to term 2
I20260810 00:46:07.573419 9230 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a2698326444a5891609266e0620d2d" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 46485 } }
I20260810 00:46:07.573911 9230 leader_election.cc:302] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93a2698326444a5891609266e0620d2d; no voters:
I20260810 00:46:07.574563 9230 leader_election.cc:288] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 00:46:07.574697 9236 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d [term 2 FOLLOWER]: Leader election won for term 2
I20260810 00:46:07.576084 9236 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d [term 2 LEADER]: Becoming Leader. State: Replica: 93a2698326444a5891609266e0620d2d, State: Running, Role: LEADER
I20260810 00:46:07.576721 9236 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a2698326444a5891609266e0620d2d" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 46485 } }
I20260810 00:46:07.576912 9230 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:46:07.579427 9244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93a2698326444a5891609266e0620d2d. Latest consensus state: current_term: 2 leader_uuid: "93a2698326444a5891609266e0620d2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a2698326444a5891609266e0620d2d" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 46485 } } }
I20260810 00:46:07.579461 9243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "93a2698326444a5891609266e0620d2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a2698326444a5891609266e0620d2d" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 46485 } } }
I20260810 00:46:07.579877 9243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d [sys.catalog]: This master's current role is: LEADER
I20260810 00:46:07.579874 9244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d [sys.catalog]: This master's current role is: LEADER
I20260810 00:46:07.585202 9249 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:46:07.592082 9249 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:46:07.593600 9249 catalog_manager.cc:1259] Loaded cluster ID: 3eae2a092cfb4684a8da771dbe1c209e
I20260810 00:46:07.593760 9249 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:46:07.597719 9249 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:46:07.599252 9249 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 93a2698326444a5891609266e0620d2d: Loaded TSK: 0
I20260810 00:46:07.600790 9249 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 00:46:07.873162 9234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:46:07.873467 9234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:46:07.873624 9234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:46:07.880777 9234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:46:07.880921 9234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:46:07.881084 9234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.1
I20260810 00:46:07.890632 9234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:43967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.1.keytab
--principal=oryx/127.28.6.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.1:36441
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.6.1
--webserver_port=41039
--tserver_master_addrs=127.28.6.62:46485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.1
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:46:07.891918 9234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:46:07.893188 9234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:46:07.907437 9276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:07.907972 9277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:07.908177 9279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:46:07.908999 9234 server_base.cc:1034] running on GCE node
I20260810 00:46:07.909708 9234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:46:07.910838 9234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:46:07.912119 9234 hybrid_clock.cc:648] HybridClock initialized: now 1786322767912048 us; error 55 us; skew 500 ppm
I20260810 00:46:07.912482 9234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:46:07 dist-test-slave-1nl6 krb5kdc[8372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322767, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.6.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:46:07.916453 9234 init.cc:377] Logged in from keytab as oryx/127.28.6.1@KRBTEST.COM (short username oryx)
I20260810 00:46:07.918388 9234 webserver.cc:460] Webserver started at http://127.28.6.1:41039/ using document root <none> and password file <none>
I20260810 00:46:07.919294 9234 fs_manager.cc:359] Metadata directory not provided
I20260810 00:46:07.919451 9234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:46:07.925607 9234 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260810 00:46:07.928310 9289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:07.929677 9234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 00:46:07.929849 9234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-0/wal
uuid: "80073accca294a6ba4a1733482669960"
format_stamp: "Formatted at 2026-08-10 00:46:05 on dist-test-slave-1nl6"
server_key: "f7ed879b337721187e18648cb83e411e"
server_key_iv: "6d9af6acc7fa496186ce491ac5416fba"
server_key_version: "encryptionkey@0"
I20260810 00:46:07.930773 9234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:46:07.946159 9234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:46:07.949867 9234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:46:07.950162 9234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:46:07.951186 9234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:46:07.952510 9234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:46:07.952625 9234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:07.952754 9234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:46:07.952855 9234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:07.987291 9234 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.1:36441
I20260810 00:46:07.987363 9449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.1:36441 every 8 connection(s)
I20260810 00:46:07.988778 9234 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
I20260810 00:46:07.996513 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 9234
I20260810 00:46:07.997363 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.2:33225
--local_ip_for_outbound_sockets=127.28.6.2
--tserver_master_addrs=127.28.6.62:46485
--webserver_port=46395
--webserver_interface=127.28.6.2
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:43967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.6.2
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.2.keytab with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 00:46:07 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.1@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:07 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.1@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:08.009078 9453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:07.991831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.1:54621)
0810 00:46:07.992665 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:07.992713 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:07.994422 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:07.995730 (+ 1308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:07.995737 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:07.996564 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:07.997364 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:07.997392 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:07.999260 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:07.999269 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:08.001372 (+ 2103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:08.001392 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:08.002423 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:08.002434 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:08.002747 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:08.003528 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:08.003563 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:08.007752 (+ 4189us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":368,"threads_started":1}
W20260810 00:46:08.009107 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:07.991754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:54621 (local address 127.28.6.62:46485)
0810 00:46:07.992459 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:07.992470 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:07.992844 (+ 374us) server_negotiation.cc:407] Connection header received
0810 00:46:07.994946 (+ 2102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:07.994971 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:07.995292 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:07.995567 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:07.997637 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:07.999097 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:08.001542 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:08.002177 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:08.002315 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:08.008132 (+ 5817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.1:54621: BlockingRecv error: recv got EOF from 127.28.6.1:54621 (error 108)
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":274,"threads_started":1}
W20260810 00:46:08.009945 9450 heartbeater.cc:644] Failed to heartbeat to 127.28.6.62:46485 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.6.62:46485: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Aug 10 00:46:08 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.1@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:08 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.1@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:08.018784 9453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:08.010619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.1:43731)
0810 00:46:08.010815 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:08.010835 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:08.010993 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:08.011322 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:08.011325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:08.011609 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:08.011740 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:08.011747 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:08.013284 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:08.013287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:08.014144 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:08.014154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:08.015180 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:08.015184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:08.015302 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:08.015895 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:08.015911 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:08.018500 (+ 2589us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260810 00:46:08.018918 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:08.010771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:43731 (local address 127.28.6.62:46485)
0810 00:46:08.011013 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:08.011021 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:08.011041 (+ 20us) server_negotiation.cc:407] Connection header received
0810 00:46:08.011099 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:08.011107 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:08.011187 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:08.011293 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:08.011893 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:08.013142 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:08.014286 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:08.015041 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:08.015201 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:08.018686 (+ 3485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.1:43731: BlockingRecv error: recv got EOF from 127.28.6.1:43731 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 10 00:46:08 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.1@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:08 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.1@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:08.027504 9453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:08.019526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.1:50365)
0810 00:46:08.019697 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:08.019712 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:08.019821 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:08.020178 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:08.020183 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:08.020476 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:08.020609 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:08.020617 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:08.021570 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:08.021576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:08.022453 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:08.022464 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:08.023278 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:08.023283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:08.023444 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:08.024045 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:08.024061 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:08.027207 (+ 3146us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260810 00:46:08.027833 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:08.019715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:50365 (local address 127.28.6.62:46485)
0810 00:46:08.019911 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:08.019917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:08.019932 (+ 15us) server_negotiation.cc:407] Connection header received
0810 00:46:08.019981 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:08.019984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:08.020052 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:08.020153 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:08.020744 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:08.021432 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:08.022599 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:08.023133 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:08.023257 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:08.027426 (+ 4169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.1:50365: BlockingRecv error: recv got EOF from 127.28.6.1:50365 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260810 00:46:08.027890 9450 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 00:46:08.342557 9454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:46:08.342810 9454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:46:08.342955 9454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:46:08.350008 9454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:46:08.350106 9454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:46:08.350229 9454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.2
I20260810 00:46:08.359670 9454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:43967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.2.keytab
--principal=oryx/127.28.6.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.2:33225
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.6.2
--webserver_port=46395
--tserver_master_addrs=127.28.6.62:46485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.2
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:46:08.360811 9454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:46:08.362018 9454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:46:08.377919 9465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:08.377871 9462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:08.379593 9461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:46:08.387910 9454 server_base.cc:1034] running on GCE node
I20260810 00:46:08.388871 9454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:46:08.390192 9454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:46:08.391501 9454 hybrid_clock.cc:648] HybridClock initialized: now 1786322768391429 us; error 52 us; skew 500 ppm
I20260810 00:46:08.392014 9454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:46:08 dist-test-slave-1nl6 krb5kdc[8372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322768, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.6.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:46:08.397639 9454 init.cc:377] Logged in from keytab as oryx/127.28.6.2@KRBTEST.COM (short username oryx)
I20260810 00:46:08.399718 9454 webserver.cc:460] Webserver started at http://127.28.6.2:46395/ using document root <none> and password file <none>
I20260810 00:46:08.400660 9454 fs_manager.cc:359] Metadata directory not provided
I20260810 00:46:08.400854 9454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:46:08.408676 9454 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.003s
I20260810 00:46:08.412209 9476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:08.414193 9454 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260810 00:46:08.414392 9454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-1/wal
uuid: "74b64516dd9549be9b73b7ca0d872a97"
format_stamp: "Formatted at 2026-08-10 00:46:05 on dist-test-slave-1nl6"
server_key: "ff5d2057199a1b208eaf0cfa8b589f09"
server_key_iv: "75a808417bafbdec32e89b866c966750"
server_key_version: "encryptionkey@0"
I20260810 00:46:08.416158 9454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:46:08.446017 9454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:46:08.450109 9454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:46:08.450354 9454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:46:08.451390 9454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:46:08.452790 9454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:46:08.452888 9454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:08.452978 9454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:46:08.453040 9454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:08.485700 9454 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.2:33225
I20260810 00:46:08.485791 9653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.2:33225 every 8 connection(s)
I20260810 00:46:08.487412 9454 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
I20260810 00:46:08.488476 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 9454
I20260810 00:46:08.489290 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.3:41379
--local_ip_for_outbound_sockets=127.28.6.3
--tserver_master_addrs=127.28.6.62:46485
--webserver_port=37017
--webserver_interface=127.28.6.3
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:43967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.6.3
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.3.keytab with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 00:46:08 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.2@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:08 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.2@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 00:46:08.511318 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:08.490297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:34979 (local address 127.28.6.62:46485)
0810 00:46:08.490518 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:08.490525 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:08.499707 (+ 9182us) server_negotiation.cc:407] Connection header received
0810 00:46:08.501571 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:08.501577 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:08.501668 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:08.501778 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:08.503369 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:08.504123 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:08.505681 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:08.506120 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:08.506254 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:08.511080 (+ 4826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.2:34979: BlockingRecv error: recv got EOF from 127.28.6.2:34979 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260810 00:46:08.512001 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:08.490829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.2:34979)
0810 00:46:08.499543 (+ 8714us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:08.499587 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:08.501292 (+ 1705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:08.502039 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:08.502048 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:08.502562 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:08.503184 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:08.503204 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:08.504296 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:08.504301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:08.505530 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:08.505543 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:08.506371 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:08.506379 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:08.506599 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:08.507315 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:08.507344 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:08.510874 (+ 3530us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":8438,"thread_start_us":8162,"threads_started":1}
W20260810 00:46:08.512684 9654 heartbeater.cc:644] Failed to heartbeat to 127.28.6.62:46485 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.6.62:46485: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Aug 10 00:46:08 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.2@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:08 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.2@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:08.524405 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:08.513345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.2:37951)
0810 00:46:08.513531 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:08.513548 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:08.513652 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:08.514289 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:08.514296 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:08.514789 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:08.515053 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:08.515068 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:08.516519 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:08.516540 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:08.518412 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:08.518432 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:08.519409 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:08.519416 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:08.519615 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:08.520236 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:08.520260 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:08.524031 (+ 3771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":70}
W20260810 00:46:08.524499 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:08.513425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:37951 (local address 127.28.6.62:46485)
0810 00:46:08.513582 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:08.513589 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:08.513606 (+ 17us) server_negotiation.cc:407] Connection header received
0810 00:46:08.513831 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:08.513839 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:08.513938 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:08.514070 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:08.515221 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:08.516353 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:08.518581 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:08.519246 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:08.519376 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:08.524241 (+ 4865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.2:37951: BlockingRecv error: recv got EOF from 127.28.6.2:37951 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 10 00:46:08 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.2@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:08 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.2@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:08.534063 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:08.525313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.2:39913)
0810 00:46:08.525549 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:08.525567 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:08.525673 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:08.525956 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:08.525962 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:08.526409 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:08.526647 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:08.526663 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:08.527664 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:08.527670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:08.529276 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:08.529292 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:08.529911 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:08.529918 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:08.530111 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:08.530675 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:08.530700 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:08.533764 (+ 3064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121}
W20260810 00:46:08.534160 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:08.525398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:39913 (local address 127.28.6.62:46485)
0810 00:46:08.525534 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:08.525538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:08.525633 (+ 95us) server_negotiation.cc:407] Connection header received
0810 00:46:08.525770 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:08.525773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:08.525824 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:08.525899 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:08.526795 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:08.527526 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:08.529424 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:08.529786 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:08.529878 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:08.533948 (+ 4070us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.2:39913: BlockingRecv error: recv got EOF from 127.28.6.2:39913 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260810 00:46:08.534508 9654 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 00:46:08.837653 9661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:46:08.837908 9661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:46:08.838001 9661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:46:08.845207 9661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:46:08.845310 9661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:46:08.845419 9661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.3
I20260810 00:46:08.854785 9661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:43967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/krb5kdc/oryx_127.28.6.3.keytab
--principal=oryx/127.28.6.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.3:41379
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.6.3
--webserver_port=37017
--tserver_master_addrs=127.28.6.62:46485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.3
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:46:08.856050 9661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:46:08.857376 9661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:46:08.870963 9671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:08.870949 9673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:08.871184 9675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:46:08.872220 9661 server_base.cc:1034] running on GCE node
I20260810 00:46:08.872816 9661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:46:08.873713 9661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:46:08.874998 9661 hybrid_clock.cc:648] HybridClock initialized: now 1786322768874942 us; error 66 us; skew 500 ppm
I20260810 00:46:08.875315 9661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:46:08 dist-test-slave-1nl6 krb5kdc[8372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322768, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.6.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:46:08.878746 9661 init.cc:377] Logged in from keytab as oryx/127.28.6.3@KRBTEST.COM (short username oryx)
I20260810 00:46:08.880254 9661 webserver.cc:460] Webserver started at http://127.28.6.3:37017/ using document root <none> and password file <none>
I20260810 00:46:08.880874 9661 fs_manager.cc:359] Metadata directory not provided
I20260810 00:46:08.880992 9661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:46:08.886122 9661 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 00:46:08.888756 9689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:08.890049 9661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 00:46:08.890239 9661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-2/wal
uuid: "7084d42358aa41bc9690ddf2b2889b80"
format_stamp: "Formatted at 2026-08-10 00:46:06 on dist-test-slave-1nl6"
server_key: "c683a9a2c0ef4c867353334664f02a0f"
server_key_iv: "997d85a52a6d0a74dec5dc4fb8ff05f3"
server_key_version: "encryptionkey@0"
I20260810 00:46:08.891245 9661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:46:08.905825 9661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:46:08.909641 9661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:46:08.909896 9661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:46:08.910827 9661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:46:08.912166 9661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:46:08.912274 9661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:08.912420 9661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:46:08.912508 9661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:08.945853 9661 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.3:41379
I20260810 00:46:08.946062 9853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.3:41379 every 8 connection(s)
I20260810 00:46:08.947470 9661 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
I20260810 00:46:08.955979 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 9661
Aug 10 00:46:08 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.3@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:08 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.3@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 00:46:08.964238 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:08.949948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:40727 (local address 127.28.6.62:46485)
0810 00:46:08.950140 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:08.950145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:08.951351 (+ 1206us) server_negotiation.cc:407] Connection header received
0810 00:46:08.953161 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:08.953167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:08.953253 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:08.953345 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:08.954837 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:08.955657 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:08.957948 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:08.958516 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:08.958616 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:08.963930 (+ 5314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.3:40727: BlockingRecv error: recv got EOF from 127.28.6.3:40727 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 00:46:08.964716 9859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:08.950354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.3:40727)
0810 00:46:08.951172 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:08.951218 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:08.952885 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:08.953565 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:08.953572 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:08.954096 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:08.954665 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:08.954681 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:08.955830 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:08.955835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:08.957807 (+ 1972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:08.957825 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:08.958786 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:08.958795 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:08.959137 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:08.959929 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:08.959960 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:08.963698 (+ 3738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":315,"threads_started":1}
W20260810 00:46:08.965440 9855 heartbeater.cc:644] Failed to heartbeat to 127.28.6.62:46485 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.6.62:46485: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Aug 10 00:46:08 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.3@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:08.972913 9861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:08.957204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39900 (local address 127.28.6.62:46485)
0810 00:46:08.957812 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:08.957816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:08.957833 (+ 17us) server_negotiation.cc:407] Connection header received
0810 00:46:08.957873 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:08.957876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:08.957967 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:08.958088 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:08.960183 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:08.960940 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:08.962311 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:08.962721 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:08.962852 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:08.965863 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:46:08.965871 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:46:08.965886 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:46:08.965922 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:46:08.969548 (+ 3626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:46:08.970309 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:46:08.970317 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:46:08.970324 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:46:08.970589 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:46:08.971005 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:46:08.971012 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:46:08.971015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:46:08.971448 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:46:08.971750 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:08.972262 (+ 512us) server_negotiation.cc:299] Negotiation successful
0810 00:46:08.972618 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":339,"threads_started":1}
Aug 10 00:46:08 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.3@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:08.974365 9859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:08.966027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.3:33319)
0810 00:46:08.966195 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:08.966218 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:08.966295 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:08.966621 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:08.966624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:08.967061 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:08.967218 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:08.967227 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:08.968600 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:08.968604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:08.969653 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:08.969665 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:08.970550 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:08.970556 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:08.970744 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:08.971302 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:08.971318 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:08.974126 (+ 2808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260810 00:46:08.974610 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:08.966108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:33319 (local address 127.28.6.62:46485)
0810 00:46:08.966275 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:08.966282 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:08.966304 (+ 22us) server_negotiation.cc:407] Connection header received
0810 00:46:08.966391 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:08.966397 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:08.966480 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:08.966598 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:08.967358 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:08.968457 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:08.969818 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:08.970416 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:08.970556 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:08.974374 (+ 3818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.3:33319: BlockingRecv error: recv got EOF from 127.28.6.3:33319 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 10 00:46:08 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.3@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:08 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.3@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:08.982852 9859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:08.975371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.3:56871)
0810 00:46:08.975530 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:08.975550 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:08.975665 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:08.976323 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:08.976328 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:08.976621 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:08.976759 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:08.976767 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:08.977781 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:08.977787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:08.978732 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:08.978743 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:08.979448 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:08.979453 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:08.979613 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:08.980127 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:08.980142 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:08.982567 (+ 2425us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 00:46:08.983035 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:08.975465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:56871 (local address 127.28.6.62:46485)
0810 00:46:08.976084 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:08.976088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:08.976102 (+ 14us) server_negotiation.cc:407] Connection header received
0810 00:46:08.976139 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:08.976142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:08.976190 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:08.976296 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:08.976902 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:08.977631 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:08.978868 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:08.979303 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:08.979427 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:08.982750 (+ 3323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.3:56871: BlockingRecv error: recv got EOF from 127.28.6.3:56871 (error 108)
Metrics: {"server-negotiator.queue_time_us":140}
W20260810 00:46:08.983228 9855 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 00:46:09 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.1@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:09 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.1@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:09.037616 9863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:09.028519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.1:41715)
0810 00:46:09.029043 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:09.029059 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:09.029217 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:09.029561 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:09.029566 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:09.030063 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:09.030265 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:09.030275 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:09.031375 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:09.031388 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:09.032372 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:09.032385 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:09.033150 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:09.033158 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:09.033327 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:09.034020 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:09.034037 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:09.037386 (+ 3349us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":274,"threads_started":1}
W20260810 00:46:09.037787 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:09.028625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:41715 (local address 127.28.6.62:46485)
0810 00:46:09.028847 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:09.028855 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:09.029149 (+ 294us) server_negotiation.cc:407] Connection header received
0810 00:46:09.029343 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:09.029348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:09.029424 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:09.029524 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:09.030412 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:09.031225 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:09.032524 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:09.032955 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:09.033085 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:09.037587 (+ 4502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.1:41715: BlockingRecv error: recv got EOF from 127.28.6.1:41715 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 10 00:46:09 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.2@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:09 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.2@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:09.543900 9865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:09.535136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.2:51787)
0810 00:46:09.535694 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:09.535710 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:09.535832 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:09.536130 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:09.536133 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:09.536592 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:09.536800 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:09.536809 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:09.537785 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:09.537789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:09.538847 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:09.538859 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:09.539723 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:09.539728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:09.539850 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:09.540373 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:09.540390 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:09.543578 (+ 3188us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":306,"threads_started":1}
W20260810 00:46:09.544052 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:09.535207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:51787 (local address 127.28.6.62:46485)
0810 00:46:09.535387 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:09.535392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:09.535809 (+ 417us) server_negotiation.cc:407] Connection header received
0810 00:46:09.535937 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:09.535941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:09.536002 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:09.536082 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:09.536925 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:09.537640 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:09.539144 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:09.539555 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:09.539654 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:09.543769 (+ 4115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.2:51787: BlockingRecv error: recv got EOF from 127.28.6.2:51787 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 10 00:46:09 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.3@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:09 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.3@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:09.995997 9866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:09.983816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.3:32925)
0810 00:46:09.984362 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:09.984379 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:09.984492 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:09.984785 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:09.984788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:09.985244 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:09.985462 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:09.985471 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:09.986611 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:09.986617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:09.987665 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:09.987683 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:09.988424 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:09.988431 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:09.988612 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:09.992732 (+ 4120us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:09.992750 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:09.995776 (+ 3026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":288,"threads_started":1}
W20260810 00:46:09.996232 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:09.983853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:32925 (local address 127.28.6.62:46485)
0810 00:46:09.984026 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:09.984031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:09.984450 (+ 419us) server_negotiation.cc:407] Connection header received
0810 00:46:09.984595 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:09.984599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:09.984660 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:09.984751 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:09.985613 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:09.986404 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:09.987855 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:09.988241 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:09.988372 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:09.995974 (+ 7602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.3:32925: BlockingRecv error: recv got EOF from 127.28.6.3:32925 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 10 00:46:10 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.1@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:10 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.1@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:10.047088 9868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:10.038658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.1:56109)
0810 00:46:10.039232 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:10.039249 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:10.039413 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:10.039773 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:10.039778 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:10.040203 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:10.040369 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:10.040378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:10.041420 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:10.041428 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:10.042344 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:10.042357 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:10.043115 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:10.043121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:10.043301 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:10.043887 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:10.043905 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:10.046785 (+ 2880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":315,"threads_started":1}
W20260810 00:46:10.047286 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:10.038720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:56109 (local address 127.28.6.62:46485)
0810 00:46:10.038982 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:10.038990 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:10.039342 (+ 352us) server_negotiation.cc:407] Connection header received
0810 00:46:10.039542 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:10.039551 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:10.039611 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:10.039723 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:10.040522 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:10.041283 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:10.042493 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:10.042955 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:10.043096 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:10.047053 (+ 3957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.1:56109: BlockingRecv error: recv got EOF from 127.28.6.1:56109 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 10 00:46:10 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.2@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:10 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.2@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:10.552848 9869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:10.544854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.2:50759)
0810 00:46:10.545412 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:10.545429 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:10.545546 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:10.545821 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:10.545824 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:10.546239 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:10.546397 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:10.546406 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:10.547430 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:10.547433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:10.548315 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:10.548336 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:10.548986 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:10.548990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:10.549097 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:10.549648 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:10.549666 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:10.552490 (+ 2824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":305,"threads_started":1}
W20260810 00:46:10.553051 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:10.544954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:50759 (local address 127.28.6.62:46485)
0810 00:46:10.545134 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:10.545139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:10.545507 (+ 368us) server_negotiation.cc:407] Connection header received
0810 00:46:10.545632 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:10.545635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:10.545695 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:10.545776 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:10.546538 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:10.547279 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:10.548455 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:10.548858 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:10.548950 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:10.552811 (+ 3861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.2:50759: BlockingRecv error: recv got EOF from 127.28.6.2:50759 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 10 00:46:10 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.3@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:10 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.3@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:11.005182 9870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:10.996810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.3:43917)
0810 00:46:10.997294 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:10.997310 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:10.997426 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:10.997843 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:10.997848 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:10.998265 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:10.998424 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:10.998432 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:10.999558 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:10.999564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:11.000499 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:11.000512 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:11.001240 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:11.001247 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:11.001425 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:11.002008 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:11.002026 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:11.004833 (+ 2807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":257,"threads_started":1}
W20260810 00:46:11.005316 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:10.997000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:43917 (local address 127.28.6.62:46485)
0810 00:46:10.997234 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:10.997242 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:10.997385 (+ 143us) server_negotiation.cc:407] Connection header received
0810 00:46:10.997551 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:10.997557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:10.997653 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:10.997791 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:10.998576 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:10.999414 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:11.000653 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:11.001082 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:11.001219 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:11.005039 (+ 3820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.3:43917: BlockingRecv error: recv got EOF from 127.28.6.3:43917 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 10 00:46:11 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.1@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:11 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.1@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:11.056522 9871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:11.048047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.1:44083)
0810 00:46:11.048579 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:11.048595 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:11.048754 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:11.049103 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:11.049109 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:11.049487 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:11.049664 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:11.049673 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:11.050684 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:11.050693 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:11.051644 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:11.051657 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:11.052367 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:11.052373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:11.052547 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:11.053128 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:11.053145 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:11.056161 (+ 3016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":273,"threads_started":1}
W20260810 00:46:11.056602 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:11.048105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:44083 (local address 127.28.6.62:46485)
0810 00:46:11.048329 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:11.048339 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:11.048681 (+ 342us) server_negotiation.cc:407] Connection header received
0810 00:46:11.048885 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:11.048891 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:11.048959 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:11.049065 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:11.049813 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:11.050540 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:11.051792 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:11.052208 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:11.052348 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:11.056361 (+ 4013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.1:44083: BlockingRecv error: recv got EOF from 127.28.6.1:44083 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 10 00:46:11 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.2@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:11 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.2@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:11.562101 9872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:11.553941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.2:50033)
0810 00:46:11.554545 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:11.554561 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:11.554677 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:11.554983 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:11.554986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:11.555337 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:11.555516 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:11.555524 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:11.556578 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:11.556581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:11.557459 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:11.557470 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:11.558140 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:11.558144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:11.558265 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:11.558827 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:11.558844 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:11.561735 (+ 2891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":322,"threads_started":1}
W20260810 00:46:11.562173 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:11.554018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:50033 (local address 127.28.6.62:46485)
0810 00:46:11.554231 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:11.554237 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:11.554644 (+ 407us) server_negotiation.cc:407] Connection header received
0810 00:46:11.554782 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:11.554786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:11.554848 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:11.554961 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:11.555669 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:11.556454 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:11.557597 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:11.558009 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:11.558125 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:11.561921 (+ 3796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.2:50033: BlockingRecv error: recv got EOF from 127.28.6.2:50033 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Aug 10 00:46:12 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.3@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:12 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.3@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:12.014401 9873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:12.006126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.3:35129)
0810 00:46:12.006711 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:12.006727 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:12.006869 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:12.007248 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:12.007253 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:12.007712 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:12.007900 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:12.007910 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:12.009127 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:12.009132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:12.010012 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:12.010024 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:12.010649 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:12.010653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:12.010784 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:12.011454 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:12.011472 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:12.014153 (+ 2681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":313,"threads_started":1}
W20260810 00:46:12.014575 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:12.006254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:35129 (local address 127.28.6.62:46485)
0810 00:46:12.006454 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:12.006459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:12.006825 (+ 366us) server_negotiation.cc:407] Connection header received
0810 00:46:12.007025 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:12.007029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:12.007092 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:12.007175 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:12.008057 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:12.008993 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:12.010136 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:12.010528 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:12.010618 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:12.014376 (+ 3758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.3:35129: BlockingRecv error: recv got EOF from 127.28.6.3:35129 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 10 00:46:12 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.1@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:12 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.1@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:12.065218 9876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:12.057418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.1:44053)
0810 00:46:12.057914 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:12.057931 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:12.058046 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:12.058330 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:12.058333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:12.058686 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:12.058862 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:12.058871 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:12.059820 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:12.059823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:12.060777 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:12.060789 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:12.061413 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:12.061417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:12.061547 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:12.062104 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:12.062121 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:12.064984 (+ 2863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":272,"threads_started":1}
W20260810 00:46:12.065428 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:12.057518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:44053 (local address 127.28.6.62:46485)
0810 00:46:12.057773 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:12.057779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:12.058007 (+ 228us) server_negotiation.cc:407] Connection header received
0810 00:46:12.058139 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:12.058143 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:12.058204 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:12.058280 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:12.058983 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:12.059705 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:12.060892 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:12.061288 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:12.061384 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:12.065188 (+ 3804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.1:44053: BlockingRecv error: recv got EOF from 127.28.6.1:44053 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Aug 10 00:46:12 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.2@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:12 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.2@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:12.572042 9877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:12.563034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.2:47799)
0810 00:46:12.563575 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:12.563592 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:12.563772 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:12.564125 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:12.564130 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:12.564533 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:12.564704 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:12.564713 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:12.565899 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:12.565905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:12.566890 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:12.566930 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:12.567678 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:12.567685 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:12.567888 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:12.568450 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:12.568467 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:12.571713 (+ 3246us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":280,"threads_started":1}
W20260810 00:46:12.572129 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:12.563107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:47799 (local address 127.28.6.62:46485)
0810 00:46:12.563384 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:12.563391 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:12.563693 (+ 302us) server_negotiation.cc:407] Connection header received
0810 00:46:12.563911 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:12.563916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:12.563982 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:12.564085 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:12.564859 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:12.565765 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:12.567062 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:12.567500 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:12.567663 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:12.571899 (+ 4236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.2:47799: BlockingRecv error: recv got EOF from 127.28.6.2:47799 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Aug 10 00:46:13 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.3@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:13 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.3@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:13.024082 9880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:13.015514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.3:58659)
0810 00:46:13.016112 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:13.016133 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:13.016284 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:13.016677 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:13.016681 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:13.017132 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:13.017296 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:13.017305 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:13.018345 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:13.018352 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:13.019381 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:13.019393 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:13.020100 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:13.020107 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:13.020296 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:13.020842 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:13.020859 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:13.023751 (+ 2892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":351,"threads_started":1}
W20260810 00:46:13.024231 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:13.015557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:58659 (local address 127.28.6.62:46485)
0810 00:46:13.015767 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:13.015774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:13.016228 (+ 454us) server_negotiation.cc:407] Connection header received
0810 00:46:13.016405 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:13.016409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:13.016514 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:13.016615 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:13.017441 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:13.018210 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:13.019522 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:13.019947 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:13.020079 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:13.023972 (+ 3893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.3:58659: BlockingRecv error: recv got EOF from 127.28.6.3:58659 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 10 00:46:13 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.1@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:13 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.1@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:13.074327 9881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:13.066147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.1:56373)
0810 00:46:13.066631 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:13.066647 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:13.066771 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:13.067088 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:13.067091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:13.067513 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:13.067669 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:13.067677 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:13.068728 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:13.068732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:13.069750 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:13.069764 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:13.070415 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:13.070419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:13.070597 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:13.071155 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:13.071172 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:13.073983 (+ 2811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":239,"threads_started":1}
W20260810 00:46:13.074433 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:13.066247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:56373 (local address 127.28.6.62:46485)
0810 00:46:13.066489 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:13.066496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:13.066725 (+ 229us) server_negotiation.cc:407] Connection header received
0810 00:46:13.066855 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:13.066859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:13.066964 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:13.067043 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:13.067814 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:13.068591 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:13.069870 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:13.070289 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:13.070390 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:13.074197 (+ 3807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.1:56373: BlockingRecv error: recv got EOF from 127.28.6.1:56373 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Aug 10 00:46:13 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.2@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:13 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.2@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:13.582149 9882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:13.573122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.2:58349)
0810 00:46:13.573707 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:13.573726 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:13.573835 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:13.574156 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:13.574160 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:13.574584 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:13.574756 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:13.574764 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:13.576079 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:13.576083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:13.577156 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:13.577171 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:13.578208 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:13.578235 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:13.578362 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:13.579093 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:13.579110 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:13.581882 (+ 2772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":297,"threads_started":1}
W20260810 00:46:13.582329 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:13.573255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:58349 (local address 127.28.6.62:46485)
0810 00:46:13.573534 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:13.573538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:13.573798 (+ 260us) server_negotiation.cc:407] Connection header received
0810 00:46:13.573937 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:13.573941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:13.574005 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:13.574101 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:13.574934 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:13.575957 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:13.577336 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:13.578048 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:13.578154 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:13.582080 (+ 3926us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.2:58349: BlockingRecv error: recv got EOF from 127.28.6.2:58349 (error 108)
Metrics: {"server-negotiator.queue_time_us":168}
Aug 10 00:46:14 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.3@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:14 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.3@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:14.033543 9883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:14.025049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.3:47023)
0810 00:46:14.025551 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:14.025567 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:14.025684 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:14.025992 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:14.025995 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:14.026426 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:14.026592 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:14.026601 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:14.027763 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:14.027770 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:14.028901 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:14.028915 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:14.029686 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:14.029692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:14.029873 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:14.030489 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:14.030508 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:14.033303 (+ 2795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":257,"threads_started":1}
W20260810 00:46:14.033740 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:14.025200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:47023 (local address 127.28.6.62:46485)
0810 00:46:14.025433 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:14.025439 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:14.025649 (+ 210us) server_negotiation.cc:407] Connection header received
0810 00:46:14.025791 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:14.025795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:14.025858 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:14.025940 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:14.026725 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:14.027616 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:14.029045 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:14.029539 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:14.029676 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:14.033525 (+ 3849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.3:47023: BlockingRecv error: recv got EOF from 127.28.6.3:47023 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Aug 10 00:46:14 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.1@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:14 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.1@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:14.084576 9884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:14.075406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.1:39617)
0810 00:46:14.075969 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:14.075991 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:14.076155 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:14.076544 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:14.076550 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:14.077040 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:14.077213 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:14.077223 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:14.078352 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:14.078357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:14.079451 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:14.079465 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:14.080203 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:14.080207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:14.080338 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:14.081032 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:14.081051 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:14.084250 (+ 3199us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":290,"threads_started":1}
W20260810 00:46:14.084720 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:14.075460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:39617 (local address 127.28.6.62:46485)
0810 00:46:14.075663 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:14.075672 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:14.076069 (+ 397us) server_negotiation.cc:407] Connection header received
0810 00:46:14.076282 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:14.076290 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:14.076373 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:14.076455 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:14.077345 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:14.078208 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:14.079594 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:14.080042 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:14.080136 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:14.084456 (+ 4320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.1:39617: BlockingRecv error: recv got EOF from 127.28.6.1:39617 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 10 00:46:14 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.2@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:14 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.2@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:14.591297 9886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:14.583162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.2:40025)
0810 00:46:14.583716 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:14.583731 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:14.583843 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:14.584087 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:14.584090 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:14.584554 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:14.584720 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:14.584729 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:14.585902 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:14.585906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:14.586918 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:14.586931 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:14.587566 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:14.587570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:14.587691 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:14.588301 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:14.588325 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:14.591027 (+ 2702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":315,"threads_started":1}
W20260810 00:46:14.591574 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:14.583189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:40025 (local address 127.28.6.62:46485)
0810 00:46:14.583378 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:14.583382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:14.583819 (+ 437us) server_negotiation.cc:407] Connection header received
0810 00:46:14.583929 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:14.583933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:14.583995 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:14.584078 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:14.584903 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:14.585778 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:14.587044 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:14.587439 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:14.587536 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:14.591283 (+ 3747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.2:40025: BlockingRecv error: recv got EOF from 127.28.6.2:40025 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 10 00:46:15 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.3@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:15 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.3@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:15.047420 9887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:15.034584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.3:39211)
0810 00:46:15.035193 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:15.035232 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:15.035381 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:15.035737 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:15.035742 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:15.036306 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:15.036471 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:15.036481 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:15.037867 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:15.037872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:15.038772 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:15.038784 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:15.039485 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:15.039490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:15.039612 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:15.044132 (+ 4520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:15.044150 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:15.047086 (+ 2936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":331,"threads_started":1}
W20260810 00:46:15.047581 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:15.034641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:39211 (local address 127.28.6.62:46485)
0810 00:46:15.034871 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:15.034880 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:15.035309 (+ 429us) server_negotiation.cc:407] Connection header received
0810 00:46:15.035530 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:15.035534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:15.035596 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:15.035694 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:15.036635 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:15.037735 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:15.038895 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:15.039338 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:15.039431 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:15.047307 (+ 7876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.3:39211: BlockingRecv error: recv got EOF from 127.28.6.3:39211 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 10 00:46:15 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.1@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:15 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.1@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:15.093887 9888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:15.085620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.1:54175)
0810 00:46:15.086299 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:15.086316 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:15.086435 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:15.086776 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:15.086779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:15.087240 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:15.087423 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:15.087432 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:15.088483 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:15.088487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:15.089390 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:15.089401 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:15.090058 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:15.090062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:15.090172 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:15.090700 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:15.090718 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:15.093518 (+ 2800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":298,"threads_started":1}
W20260810 00:46:15.094019 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:15.085678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:54175 (local address 127.28.6.62:46485)
0810 00:46:15.085852 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:15.085857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:15.086410 (+ 553us) server_negotiation.cc:407] Connection header received
0810 00:46:15.086536 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:15.086540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:15.086602 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:15.086762 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:15.087576 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:15.088361 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:15.089508 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:15.089934 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:15.090025 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:15.093716 (+ 3691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.1:54175: BlockingRecv error: recv got EOF from 127.28.6.1:54175 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 10 00:46:15 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.2@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:15 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.2@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:15.601081 9891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:15.592311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.2:35151)
0810 00:46:15.592894 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:15.592910 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:15.593065 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:15.593393 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:15.593397 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:15.593838 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:15.594035 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:15.594050 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:15.595211 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:15.595216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:15.596123 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:15.596135 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:15.596959 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:15.596965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:15.597139 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:15.597808 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:15.597837 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:15.600726 (+ 2889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":243,"threads_started":1}
W20260810 00:46:15.601145 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:15.592383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:35151 (local address 127.28.6.62:46485)
0810 00:46:15.592595 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:15.592603 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:15.592995 (+ 392us) server_negotiation.cc:407] Connection header received
0810 00:46:15.593185 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:15.593196 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:15.593256 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:15.593364 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:15.594206 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:15.595068 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:15.596267 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:15.596797 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:15.596943 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:15.600899 (+ 3956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.2:35151: BlockingRecv error: recv got EOF from 127.28.6.2:35151 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 10 00:46:16 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.3@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:16 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.3@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:16.056849 9893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:16.048448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.3:37489)
0810 00:46:16.049077 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:16.049094 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:16.049211 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:16.049515 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:16.049518 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:16.049930 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:16.050121 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:16.050130 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:16.051118 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:16.051122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:16.052031 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:16.052043 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:16.052726 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:16.052730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:16.052835 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:16.053463 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:16.053500 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:16.056595 (+ 3095us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":286,"threads_started":1}
W20260810 00:46:16.056989 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:16.048528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:37489 (local address 127.28.6.62:46485)
0810 00:46:16.048690 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:16.048695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:16.049165 (+ 470us) server_negotiation.cc:407] Connection header received
0810 00:46:16.049315 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:16.049318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:16.049386 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:16.049467 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:16.050242 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:16.051005 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:16.052160 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:16.052590 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:16.052682 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:16.056781 (+ 4099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.3:37489: BlockingRecv error: recv got EOF from 127.28.6.3:37489 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 10 00:46:16 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.1@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:16 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.1@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:16.103744 9895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:16.094972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.1:50375)
0810 00:46:16.095561 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:16.095577 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:16.095736 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:16.096075 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:16.096081 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:16.096531 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:16.096690 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:16.096699 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:16.097733 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:16.097739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:16.098670 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:16.098681 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:16.099521 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:16.099531 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:16.099705 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:16.100403 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:16.100430 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:16.103398 (+ 2968us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":376,"threads_started":1}
W20260810 00:46:16.103850 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:16.095124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:50375 (local address 127.28.6.62:46485)
0810 00:46:16.095345 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:16.095352 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:16.095658 (+ 306us) server_negotiation.cc:407] Connection header received
0810 00:46:16.095863 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:16.095873 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:16.095933 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:16.096038 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:16.096846 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:16.097582 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:16.098836 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:16.099330 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:16.099456 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:16.103606 (+ 4150us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.1:50375: BlockingRecv error: recv got EOF from 127.28.6.1:50375 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 10 00:46:16 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.2@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:16 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.2@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:16.610595 9896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:16.602130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.2:56309)
0810 00:46:16.602650 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:16.602667 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:16.602825 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:16.603192 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:16.603197 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:16.603632 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:16.603783 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:16.603791 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:16.604979 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:16.604985 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:16.605907 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:16.605919 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:16.606699 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:16.606706 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:16.606877 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:16.607510 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:16.607528 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:16.610366 (+ 2838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":416,"threads_started":1}
W20260810 00:46:16.610775 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:16.602373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:56309 (local address 127.28.6.62:46485)
0810 00:46:16.602543 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:16.602550 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:16.602747 (+ 197us) server_negotiation.cc:407] Connection header received
0810 00:46:16.602965 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:16.602976 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:16.603036 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:16.603137 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:16.603940 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:16.604831 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:16.606056 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:16.606502 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:16.606630 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:16.610544 (+ 3914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.2:56309: BlockingRecv error: recv got EOF from 127.28.6.2:56309 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Aug 10 00:46:17 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.3@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:17 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.3@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:17.065914 9897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:17.057903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.3:46053)
0810 00:46:17.058433 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:17.058451 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:17.058569 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:17.058883 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:17.058886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:17.059332 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:17.059497 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:17.059506 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:17.060515 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:17.060519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:17.061430 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:17.061441 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:17.062100 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:17.062104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:17.062222 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:17.062793 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:17.062810 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:17.065683 (+ 2873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":293,"threads_started":1}
W20260810 00:46:17.066174 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:17.057949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:46053 (local address 127.28.6.62:46485)
0810 00:46:17.058149 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:17.058153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:17.058524 (+ 371us) server_negotiation.cc:407] Connection header received
0810 00:46:17.058673 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:17.058678 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:17.058749 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:17.058831 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:17.059623 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:17.060373 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:17.061550 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:17.061948 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:17.062037 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:17.065892 (+ 3855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.3:46053: BlockingRecv error: recv got EOF from 127.28.6.3:46053 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 10 00:46:17 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.1@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:17 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.1@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:17.112618 9900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:17.104668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.1:37115)
0810 00:46:17.105219 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:17.105237 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:17.105350 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:17.105627 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:17.105630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:17.106013 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:17.106202 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:17.106211 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:17.107190 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:17.107194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:17.108135 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:17.108147 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:17.108811 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:17.108815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:17.108932 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:17.109479 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:17.109496 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:17.112376 (+ 2880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":290,"threads_started":1}
W20260810 00:46:17.112773 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:17.104733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:37115 (local address 127.28.6.62:46485)
0810 00:46:17.104890 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:17.104894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:17.105307 (+ 413us) server_negotiation.cc:407] Connection header received
0810 00:46:17.105432 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:17.105436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:17.105506 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:17.105586 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:17.106350 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:17.107079 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:17.108258 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:17.108671 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:17.108784 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:17.112560 (+ 3776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.1:37115: BlockingRecv error: recv got EOF from 127.28.6.1:37115 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 10 00:46:17 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.2@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:17 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.2@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:17.619973 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:17.611685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.2:46497)
0810 00:46:17.612214 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:17.612230 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:17.612384 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:17.612716 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:17.612721 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:17.613142 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:17.613310 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:17.613319 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:17.614342 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:17.614350 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:17.615311 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:17.615324 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:17.616035 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:17.616041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:17.616216 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:17.616815 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:17.616842 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:17.619618 (+ 2776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":278,"threads_started":1}
W20260810 00:46:17.620041 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:17.611743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:46497 (local address 127.28.6.62:46485)
0810 00:46:17.611956 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:17.611966 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:17.612317 (+ 351us) server_negotiation.cc:407] Connection header received
0810 00:46:17.612513 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:17.612519 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:17.612577 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:17.612677 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:17.613462 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:17.614199 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:17.615456 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:17.615866 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:17.615994 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:17.619820 (+ 3826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.2:46497: BlockingRecv error: recv got EOF from 127.28.6.2:46497 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 10 00:46:18 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.3@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:18 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.3@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:18.075920 9903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:18.067009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.3:40909)
0810 00:46:18.067549 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:18.067566 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:18.067691 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:18.068035 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:18.068039 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:18.068486 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:18.068671 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:18.068681 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:18.069752 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:18.069757 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:18.070773 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:18.070804 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:18.071632 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:18.071639 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:18.071821 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:18.072519 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:18.072546 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:18.075683 (+ 3137us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":309,"threads_started":1}
W20260810 00:46:18.076081 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:18.067018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:40909 (local address 127.28.6.62:46485)
0810 00:46:18.067225 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:18.067234 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:18.067644 (+ 410us) server_negotiation.cc:407] Connection header received
0810 00:46:18.067803 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:18.067807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:18.067868 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:18.067994 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:18.068815 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:18.069605 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:18.070967 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:18.071454 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:18.071588 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:18.075879 (+ 4291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.3:40909: BlockingRecv error: recv got EOF from 127.28.6.3:40909 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 10 00:46:18 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.1@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:18 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.1@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:18.121985 9904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:18.113535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.1:33771)
0810 00:46:18.114115 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:18.114132 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:18.114249 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:18.114562 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:18.114565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:18.114960 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:18.115109 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:18.115119 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:18.116099 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:18.116103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:18.117052 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:18.117064 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:18.117805 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:18.117828 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:18.118016 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:18.118770 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:18.118798 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:18.121767 (+ 2969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":340,"threads_started":1}
W20260810 00:46:18.122193 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:18.113605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:33771 (local address 127.28.6.62:46485)
0810 00:46:18.113756 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:18.113761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:18.114198 (+ 437us) server_negotiation.cc:407] Connection header received
0810 00:46:18.114379 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:18.114382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:18.114440 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:18.114514 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:18.115232 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:18.115974 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:18.117173 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:18.117636 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:18.117782 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:18.121943 (+ 4161us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.1:33771: BlockingRecv error: recv got EOF from 127.28.6.1:33771 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 10 00:46:18 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.2@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:18 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.2@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:18.630087 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:18.621154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.2:52511)
0810 00:46:18.621754 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:18.621770 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:18.621876 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:18.622272 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:18.622276 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:18.623045 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:18.623269 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:18.623284 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:18.624444 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:18.624449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:18.625360 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:18.625372 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:18.626072 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:18.626079 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:18.626277 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:18.626838 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:18.626855 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:18.629740 (+ 2885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":343,"threads_started":1}
W20260810 00:46:18.630211 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:18.621252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:52511 (local address 127.28.6.62:46485)
0810 00:46:18.621412 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:18.621417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:18.621836 (+ 419us) server_negotiation.cc:407] Connection header received
0810 00:46:18.622035 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:18.622040 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:18.622098 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:18.622178 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:18.623432 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:18.624301 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:18.625519 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:18.625922 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:18.626052 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:18.629934 (+ 3882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.2:52511: BlockingRecv error: recv got EOF from 127.28.6.2:52511 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 10 00:46:19 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.3@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:19 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.3@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:19.085402 9906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:19.076835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.3:41683)
0810 00:46:19.077433 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:19.077465 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:19.077574 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:19.077867 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:19.077873 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:19.078255 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:19.078416 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:19.078425 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:19.079475 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:19.079479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:19.080414 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:19.080425 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:19.081299 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:19.081305 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:19.081514 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:19.082250 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:19.082279 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:19.085189 (+ 2910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":315,"threads_started":1}
W20260810 00:46:19.085626 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:19.076962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:41683 (local address 127.28.6.62:46485)
0810 00:46:19.077149 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:19.077156 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:19.077536 (+ 380us) server_negotiation.cc:407] Connection header received
0810 00:46:19.077667 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:19.077672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:19.077740 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:19.077823 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:19.078544 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:19.079346 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:19.080553 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:19.081120 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:19.081268 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:19.085367 (+ 4099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.3:41683: BlockingRecv error: recv got EOF from 127.28.6.3:41683 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 10 00:46:19 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.1@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:19 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.1@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:19.131749 9907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:19.122940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.1:46979)
0810 00:46:19.123457 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:19.123479 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:19.123629 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:19.123988 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:19.123995 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:19.124446 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:19.124625 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:19.124635 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:19.125663 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:19.125670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:19.126585 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:19.126596 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:19.127409 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:19.127415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:19.127598 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:19.128306 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:19.128329 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:19.131490 (+ 3161us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":248,"threads_started":1}
W20260810 00:46:19.131892 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:19.122993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:46979 (local address 127.28.6.62:46485)
0810 00:46:19.123242 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:19.123251 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:19.123573 (+ 322us) server_negotiation.cc:407] Connection header received
0810 00:46:19.123775 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:19.123781 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:19.123851 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:19.123952 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:19.124779 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:19.125526 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:19.126731 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:19.127250 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:19.127376 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:19.131688 (+ 4312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.1:46979: BlockingRecv error: recv got EOF from 127.28.6.1:46979 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 10 00:46:19 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.2@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:19 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.2@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:19.639631 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:19.631114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.2:46095)
0810 00:46:19.631700 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:19.631731 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:19.631883 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:19.632353 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:19.632360 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:19.632776 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:19.632963 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:19.632971 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:19.633951 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:19.633955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:19.634981 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:19.634993 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:19.635680 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:19.635685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:19.635801 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:19.636389 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:19.636406 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:19.639278 (+ 2872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":340,"threads_started":1}
W20260810 00:46:19.639756 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:19.631158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:46095 (local address 127.28.6.62:46485)
0810 00:46:19.631375 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:19.631385 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:19.631804 (+ 419us) server_negotiation.cc:407] Connection header received
0810 00:46:19.632021 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:19.632026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:19.632113 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:19.632249 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:19.633084 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:19.633837 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:19.635097 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:19.635501 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:19.635660 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:19.639509 (+ 3849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.2:46095: BlockingRecv error: recv got EOF from 127.28.6.2:46095 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 10 00:46:20 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.3@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:20 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.3@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:20.099720 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:20.086506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.3:46555)
0810 00:46:20.087055 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:20.087071 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:20.087224 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:20.087571 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:20.087575 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:20.088026 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:20.088200 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:20.088210 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:20.089443 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:20.089448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:20.090345 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:20.090356 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:20.091066 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:20.091072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:20.091259 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:20.096138 (+ 4879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:20.096157 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:20.099475 (+ 3318us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":265,"threads_started":1}
W20260810 00:46:20.099850 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:20.086569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:46555 (local address 127.28.6.62:46485)
0810 00:46:20.086770 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:20.086777 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:20.087171 (+ 394us) server_negotiation.cc:407] Connection header received
0810 00:46:20.087372 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:20.087376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:20.087436 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:20.087535 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:20.088350 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:20.089311 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:20.090481 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:20.090878 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:20.091048 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:20.099645 (+ 8597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.3:46555: BlockingRecv error: recv got EOF from 127.28.6.3:46555 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 10 00:46:20 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.1@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:20 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.1@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:20.143383 9911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:20.132848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.1:37767)
0810 00:46:20.133341 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:20.133357 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:20.133472 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:20.133774 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:20.133778 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:20.134217 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:20.134377 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:20.134387 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:20.135517 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:20.135523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:20.136899 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:20.136913 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:20.137724 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:20.137732 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:20.137978 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:20.138759 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:20.138786 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:20.142888 (+ 4102us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":270,"threads_started":1}
W20260810 00:46:20.143792 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:20.132872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:37767 (local address 127.28.6.62:46485)
0810 00:46:20.133116 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:20.133125 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:20.133440 (+ 315us) server_negotiation.cc:407] Connection header received
0810 00:46:20.133567 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:20.133571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:20.133630 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:20.133707 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:20.134512 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:20.135352 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:20.137032 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:20.137570 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:20.137663 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:20.143358 (+ 5695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.1:37767: BlockingRecv error: recv got EOF from 127.28.6.1:37767 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 10 00:46:20 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.2@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:20 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.2@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:20.648500 9912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:20.640476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.2:55241)
0810 00:46:20.640966 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:20.640983 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:20.641104 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:20.641394 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:20.641398 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:20.641768 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:20.641922 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:20.641930 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:20.642995 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:20.642998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:20.644038 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:20.644052 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:20.644781 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:20.644785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:20.644918 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:20.645508 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:20.645524 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:20.648259 (+ 2735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":286,"threads_started":1}
W20260810 00:46:20.648672 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:20.640538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:55241 (local address 127.28.6.62:46485)
0810 00:46:20.640792 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:20.640799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:20.641060 (+ 261us) server_negotiation.cc:407] Connection header received
0810 00:46:20.641196 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:20.641200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:20.641275 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:20.641353 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:20.642074 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:20.642824 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:20.644229 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:20.644656 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:20.644755 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:20.648466 (+ 3711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.2:55241: BlockingRecv error: recv got EOF from 127.28.6.2:55241 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 10 00:46:21 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.3@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:21 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.3@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:21.109014 9913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:21.100724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.3:39343)
0810 00:46:21.101217 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:21.101233 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:21.101407 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:21.101742 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:21.101747 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:21.102164 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:21.102317 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:21.102335 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:21.103417 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:21.103423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:21.104344 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:21.104355 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:21.105044 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:21.105050 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:21.105241 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:21.105820 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:21.105840 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:21.108678 (+ 2838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":281,"threads_started":1}
W20260810 00:46:21.109149 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:21.100794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:39343 (local address 127.28.6.62:46485)
0810 00:46:21.101033 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:21.101042 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:21.101314 (+ 272us) server_negotiation.cc:407] Connection header received
0810 00:46:21.101542 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:21.101547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:21.101607 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:21.101704 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:21.102468 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:21.103266 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:21.104485 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:21.104903 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:21.105030 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:21.108896 (+ 3866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.3:39343: BlockingRecv error: recv got EOF from 127.28.6.3:39343 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 00:46:21 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.1@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:21 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.1@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:21.152835 9914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:21.144657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.1:39257)
0810 00:46:21.145243 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:21.145259 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:21.145413 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:21.145787 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:21.145791 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:21.146195 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:21.146379 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:21.146388 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:21.147343 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:21.147347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:21.148266 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:21.148277 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:21.149117 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:21.149121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:21.149241 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:21.149787 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:21.149804 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:21.152587 (+ 2783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":260,"threads_started":1}
W20260810 00:46:21.152972 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:21.144688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:39257 (local address 127.28.6.62:46485)
0810 00:46:21.144911 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:21.144920 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:21.145360 (+ 440us) server_negotiation.cc:407] Connection header received
0810 00:46:21.145553 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:21.145558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:21.145639 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:21.145733 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:21.146503 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:21.147233 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:21.148408 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:21.148950 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:21.149078 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:21.152753 (+ 3675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.1:39257: BlockingRecv error: recv got EOF from 127.28.6.1:39257 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 10 00:46:21 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.2@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:21 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.2@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:21.660616 9915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:21.649540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.2:38683)
0810 00:46:21.650481 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:21.650503 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:21.650661 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:21.651057 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:21.651064 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:21.651688 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:21.651980 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:21.651996 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:21.653252 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:21.653258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:21.654883 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:21.654927 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:21.656259 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:21.656265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:21.656445 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:21.657150 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:21.657173 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:21.660378 (+ 3205us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":788,"spinlock_wait_cycles":35712,"thread_start_us":760,"threads_started":1}
W20260810 00:46:21.661044 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:21.649751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:38683 (local address 127.28.6.62:46485)
0810 00:46:21.649972 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:21.649993 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:21.650594 (+ 601us) server_negotiation.cc:407] Connection header received
0810 00:46:21.650808 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:21.650813 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:21.650878 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:21.651009 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:21.652168 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:21.653053 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:21.655085 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:21.655836 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:21.655938 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:21.660738 (+ 4800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.2:38683: BlockingRecv error: recv got EOF from 127.28.6.2:38683 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 10 00:46:22 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.3@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:22 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.3@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:22.118507 9916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:22.110191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.3:45583)
0810 00:46:22.110764 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:22.110792 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:22.110964 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:22.111330 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:22.111337 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:22.111774 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:22.111954 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:22.111964 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:22.112975 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:22.112981 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:22.113901 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:22.113912 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:22.114665 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:22.114671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:22.114846 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:22.115506 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:22.115524 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:22.118272 (+ 2748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":276,"threads_started":1}
W20260810 00:46:22.118698 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:22.110231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:45583 (local address 127.28.6.62:46485)
0810 00:46:22.110447 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:22.110454 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:22.110879 (+ 425us) server_negotiation.cc:407] Connection header received
0810 00:46:22.111108 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:22.111114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:22.111180 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:22.111281 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:22.112106 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:22.112832 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:22.114051 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:22.114489 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:22.114618 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:22.118471 (+ 3853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.3:45583: BlockingRecv error: recv got EOF from 127.28.6.3:45583 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 10 00:46:22 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.1@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:22 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.1@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:22.162524 9917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:22.153899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.1:38599)
0810 00:46:22.154451 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:22.154468 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:22.154610 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:22.154946 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:22.154951 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:22.155306 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:22.155466 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:22.155475 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:22.156455 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:22.156459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:22.157670 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:22.157683 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:22.158620 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:22.158626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:22.158846 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:22.159514 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:22.159532 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:22.162274 (+ 2742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":274,"threads_started":1}
W20260810 00:46:22.162715 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:22.153979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:38599 (local address 127.28.6.62:46485)
0810 00:46:22.154312 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:22.154318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:22.154550 (+ 232us) server_negotiation.cc:407] Connection header received
0810 00:46:22.154731 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:22.154735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:22.154809 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:22.154892 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:22.155587 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:22.156325 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:22.157814 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:22.158482 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:22.158606 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:22.162472 (+ 3866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.1:38599: BlockingRecv error: recv got EOF from 127.28.6.1:38599 (error 108)
Metrics: {"server-negotiator.queue_time_us":193}
Aug 10 00:46:22 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.2@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:22 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.2@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:22.669965 9918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:22.661933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.2:51757)
0810 00:46:22.662458 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:22.662482 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:22.662596 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:22.662943 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:22.662952 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:22.663385 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:22.663546 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:22.663555 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:22.664544 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:22.664548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:22.665465 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:22.665476 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:22.666097 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:22.666101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:22.666239 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:22.666821 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:22.666838 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:22.669716 (+ 2878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":291,"threads_started":1}
W20260810 00:46:22.670132 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:22.662019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:51757 (local address 127.28.6.62:46485)
0810 00:46:22.662181 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:22.662186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:22.662551 (+ 365us) server_negotiation.cc:407] Connection header received
0810 00:46:22.662686 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:22.662690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:22.662750 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:22.662831 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:22.663669 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:22.664417 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:22.665583 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:22.665985 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:22.666073 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:22.669892 (+ 3819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.2:51757: BlockingRecv error: recv got EOF from 127.28.6.2:51757 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 10 00:46:23 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.3@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:23 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.3@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:23.127797 9919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:23.119428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.3:48305)
0810 00:46:23.119892 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:23.119909 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:23.120027 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:23.120331 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:23.120334 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:23.120702 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:23.120857 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:23.120865 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:23.121805 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:23.121809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:23.122958 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:23.122971 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:23.123685 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:23.123690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:23.123816 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:23.124453 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:23.124471 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:23.127428 (+ 2957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":259,"threads_started":1}
W20260810 00:46:23.127923 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:23.119472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:48305 (local address 127.28.6.62:46485)
0810 00:46:23.119699 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:23.119708 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:23.119990 (+ 282us) server_negotiation.cc:407] Connection header received
0810 00:46:23.120130 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:23.120134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:23.120205 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:23.120283 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:23.120976 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:23.121675 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:23.123127 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:23.123537 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:23.123629 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:23.127680 (+ 4051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.3:48305: BlockingRecv error: recv got EOF from 127.28.6.3:48305 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 10 00:46:23 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.1@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:23 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.1@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:23.171653 9922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:23.163656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.1:52347)
0810 00:46:23.164146 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:23.164162 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:23.164299 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:23.164599 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:23.164602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:23.164985 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:23.165149 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:23.165158 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:23.166089 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:23.166094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:23.167095 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:23.167107 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:23.167788 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:23.167792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:23.167914 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:23.168467 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:23.168484 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:23.171405 (+ 2921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":263,"threads_started":1}
W20260810 00:46:23.171793 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:23.163680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:52347 (local address 127.28.6.62:46485)
0810 00:46:23.163897 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:23.163907 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:23.164238 (+ 331us) server_negotiation.cc:407] Connection header received
0810 00:46:23.164394 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:23.164398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:23.164470 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:23.164560 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:23.165293 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:23.165972 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:23.167210 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:23.167656 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:23.167746 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:23.171583 (+ 3837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.1:52347: BlockingRecv error: recv got EOF from 127.28.6.1:52347 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 10 00:46:23 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.2@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 00:46:23 dist-test-slave-1nl6 krb5kdc[8372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.6.2@KRBTEST.COM for oryx/127.28.6.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 00:46:23.679800 9923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:23.671191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:46485 (local address 127.28.6.2:47157)
0810 00:46:23.671682 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:23.671698 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:23.671806 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:23.672143 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:23.672161 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:23.672774 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:23.672972 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:23.672981 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:23.673930 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:23.673933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:23.674880 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:23.674892 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:23.675834 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:23.675839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:23.675989 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:23.676659 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:23.676676 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:23.679546 (+ 2870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.6.62:46485: Server oryx/127.28.6.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":271,"threads_started":1}
W20260810 00:46:23.679939 9452 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:23.671251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:47157 (local address 127.28.6.62:46485)
0810 00:46:23.671489 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:23.671496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:23.671774 (+ 278us) server_negotiation.cc:407] Connection header received
0810 00:46:23.671919 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:23.671924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:23.671986 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:23.672067 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:23.673091 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:23.673805 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:23.675056 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:23.675694 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:23.675830 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:23.679733 (+ 3903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.6.2:47157: BlockingRecv error: recv got EOF from 127.28.6.2:47157 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 00:46:23.967152 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 9234
I20260810 00:46:23.979290 9926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:23.968313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60364 (local address 127.28.6.1:36441)
0810 00:46:23.968892 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:23.968899 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:23.968924 (+ 25us) server_negotiation.cc:407] Connection header received
0810 00:46:23.969023 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:23.969034 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:23.969271 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:23.969443 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:23.971021 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:23.972049 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:23.973214 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:23.973750 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:23.973855 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:23.976353 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:46:23.976362 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:46:23.976375 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:46:23.976405 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:46:23.978762 (+ 2357us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.6.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0810 00:46:23.979071 (+ 309us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60364: Request ticket server kudu/127.28.6.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":281,"threads_started":1}
W20260810 00:46:23.979423 9926 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60364: Request ticket server kudu/127.28.6.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260810 00:46:23.979435 28696 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 9234: Not authorized: Client connection negotiation failed: client connection to 127.28.6.1:36441: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.6.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260810 00:46:23.979688 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 9234
I20260810 00:46:24.007208 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 9454
I20260810 00:46:24.018791 9927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:24.008149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55540 (local address 127.28.6.2:33225)
0810 00:46:24.008760 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:24.008768 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:24.008800 (+ 32us) server_negotiation.cc:407] Connection header received
0810 00:46:24.008866 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:24.008872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:24.009086 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:24.009276 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:24.010422 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:24.011581 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:24.012751 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:24.013291 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:24.013387 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:24.015759 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:46:24.015769 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:46:24.015782 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:46:24.015817 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:46:24.018276 (+ 2459us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.6.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0810 00:46:24.018510 (+ 234us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55540: Request ticket server kudu/127.28.6.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":297,"threads_started":1}
W20260810 00:46:24.018923 9927 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55540: Request ticket server kudu/127.28.6.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260810 00:46:24.018941 28696 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 9454: Not authorized: Client connection negotiation failed: client connection to 127.28.6.2:33225: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.6.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260810 00:46:24.019122 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 9454
I20260810 00:46:24.056064 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 9661
I20260810 00:46:24.068041 9928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:24.056995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33826 (local address 127.28.6.3:41379)
0810 00:46:24.057632 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:24.057639 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:24.057667 (+ 28us) server_negotiation.cc:407] Connection header received
0810 00:46:24.057732 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:24.057739 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:24.057976 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:24.058142 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:24.059535 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:24.060586 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:24.061733 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:24.062304 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:24.062430 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:24.064957 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:46:24.064966 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:46:24.064980 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:46:24.065019 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:46:24.067355 (+ 2336us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.6.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0810 00:46:24.067667 (+ 312us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33826: Request ticket server kudu/127.28.6.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":316,"threads_started":1}
W20260810 00:46:24.068013 28696 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 9661: Not authorized: Client connection negotiation failed: client connection to 127.28.6.3:41379: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.6.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260810 00:46:24.068141 9928 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33826: Request ticket server kudu/127.28.6.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260810 00:46:24.068202 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 9661
I20260810 00:46:24.099953 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 9131
I20260810 00:46:24.101300 9223 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:46:24.222617 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 9131
2026-08-10T00:46:24Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21455 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-10T00:46:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:46:24Z Disabled control of system clock
I20260810 00:46:24.315845 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:36021
--webserver_interface=127.28.6.62
--webserver_port=0
--builtin_ntp_servers=127.28.6.20:44313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:36021
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 00:46:24.649560 9933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:46:24.649827 9933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:46:24.649955 9933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:46:24.656982 9933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:46:24.657074 9933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:46:24.657166 9933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:46:24.657275 9933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:46:24.657338 9933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:46:24.667644 9933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:44313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:36021
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:36021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:46:24.668843 9933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:46:24.670588 9933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:46:24.682089 9940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:24.682120 9944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:24.682147 9941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:46:24.683180 9933 server_base.cc:1034] running on GCE node
I20260810 00:46:24.683981 9933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:46:24.685304 9933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:46:24.686542 9933 hybrid_clock.cc:648] HybridClock initialized: now 1786322784686489 us; error 51 us; skew 500 ppm
I20260810 00:46:24.686946 9933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 00:46:24.689033 9933 webserver.cc:460] Webserver started at http://127.28.6.62:37273/ using document root <none> and password file <none>
I20260810 00:46:24.689668 9933 fs_manager.cc:359] Metadata directory not provided
I20260810 00:46:24.689782 9933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:46:24.690119 9933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:46:24.692991 9933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "6c413922be204a2182a4ae56c9d6976b"
format_stamp: "Formatted at 2026-08-10 00:46:24 on dist-test-slave-1nl6"
server_key: "8a61697dde5bf70b4679c84ab77fe493"
server_key_iv: "84c60e5321933e3b3d42cdeea2d569e6"
server_key_version: "encryptionkey@0"
I20260810 00:46:24.693780 9933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "6c413922be204a2182a4ae56c9d6976b"
format_stamp: "Formatted at 2026-08-10 00:46:24 on dist-test-slave-1nl6"
server_key: "8a61697dde5bf70b4679c84ab77fe493"
server_key_iv: "84c60e5321933e3b3d42cdeea2d569e6"
server_key_version: "encryptionkey@0"
I20260810 00:46:24.698766 9933 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260810 00:46:24.701984 9952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:24.703265 9933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 00:46:24.703452 9933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "6c413922be204a2182a4ae56c9d6976b"
format_stamp: "Formatted at 2026-08-10 00:46:24 on dist-test-slave-1nl6"
server_key: "8a61697dde5bf70b4679c84ab77fe493"
server_key_iv: "84c60e5321933e3b3d42cdeea2d569e6"
server_key_version: "encryptionkey@0"
I20260810 00:46:24.703665 9933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:46:24.729477 9933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:46:24.730301 9933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:46:24.730592 9933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:46:24.749104 9933 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:36021
I20260810 00:46:24.749125 10019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:36021 every 8 connection(s)
I20260810 00:46:24.750463 9933 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:46:24.754169 10020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:46:24.756763 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 9933
I20260810 00:46:24.756978 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:46:24.757689 28696 external_mini_cluster.cc:1428] Setting key a04b4357f471dd216c53e2609d55ceb9
I20260810 00:46:24.767417 10020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c413922be204a2182a4ae56c9d6976b: Bootstrap starting.
I20260810 00:46:24.771133 10020 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c413922be204a2182a4ae56c9d6976b: Neither blocks nor log segments found. Creating new log.
I20260810 00:46:24.772287 10020 log.cc:824] T 00000000000000000000000000000000 P 6c413922be204a2182a4ae56c9d6976b: Log is configured to *not* fsync() on all Append() calls
I20260810 00:46:24.772701 10027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:24.759844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59836 (local address 127.28.6.62:36021)
0810 00:46:24.760920 (+ 1076us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:24.760944 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:24.760995 (+ 51us) server_negotiation.cc:407] Connection header received
0810 00:46:24.762944 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:24.762995 (+ 51us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:24.763615 (+ 620us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:24.764050 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:24.764594 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:24.766471 (+ 1877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:24.767903 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:24.768882 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:24.769096 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:24.770209 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:46:24.770219 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:46:24.770238 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:46:24.770930 (+ 692us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:46:24.771052 (+ 122us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:46:24.771081 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:46:24.771211 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:24.771486 (+ 275us) server_negotiation.cc:299] Negotiation successful
0810 00:46:24.771765 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":306,"threads_started":1}
I20260810 00:46:24.775655 10020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c413922be204a2182a4ae56c9d6976b: No bootstrap required, opened a new log
I20260810 00:46:24.783003 10020 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c413922be204a2182a4ae56c9d6976b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c413922be204a2182a4ae56c9d6976b" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36021 } }
I20260810 00:46:24.783370 10020 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c413922be204a2182a4ae56c9d6976b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:46:24.783444 10020 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c413922be204a2182a4ae56c9d6976b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c413922be204a2182a4ae56c9d6976b, State: Initialized, Role: FOLLOWER
I20260810 00:46:24.783900 10020 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c413922be204a2182a4ae56c9d6976b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c413922be204a2182a4ae56c9d6976b" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36021 } }
I20260810 00:46:24.784081 10020 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c413922be204a2182a4ae56c9d6976b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:46:24.784174 10020 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c413922be204a2182a4ae56c9d6976b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:46:24.784386 10020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c413922be204a2182a4ae56c9d6976b [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:46:24.786732 10020 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c413922be204a2182a4ae56c9d6976b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c413922be204a2182a4ae56c9d6976b" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36021 } }
I20260810 00:46:24.787262 10020 leader_election.cc:302] T 00000000000000000000000000000000 P 6c413922be204a2182a4ae56c9d6976b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c413922be204a2182a4ae56c9d6976b; no voters:
I20260810 00:46:24.787760 10020 leader_election.cc:288] T 00000000000000000000000000000000 P 6c413922be204a2182a4ae56c9d6976b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:46:24.787930 10031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c413922be204a2182a4ae56c9d6976b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:46:24.788630 10031 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c413922be204a2182a4ae56c9d6976b [term 1 LEADER]: Becoming Leader. State: Replica: 6c413922be204a2182a4ae56c9d6976b, State: Running, Role: LEADER
I20260810 00:46:24.789166 10031 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c413922be204a2182a4ae56c9d6976b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c413922be204a2182a4ae56c9d6976b" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36021 } }
I20260810 00:46:24.789721 10020 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c413922be204a2182a4ae56c9d6976b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:46:24.792263 10033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c413922be204a2182a4ae56c9d6976b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c413922be204a2182a4ae56c9d6976b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c413922be204a2182a4ae56c9d6976b" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36021 } } }
I20260810 00:46:24.792744 10033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c413922be204a2182a4ae56c9d6976b [sys.catalog]: This master's current role is: LEADER
I20260810 00:46:24.792269 10034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c413922be204a2182a4ae56c9d6976b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c413922be204a2182a4ae56c9d6976b. Latest consensus state: current_term: 1 leader_uuid: "6c413922be204a2182a4ae56c9d6976b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c413922be204a2182a4ae56c9d6976b" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36021 } } }
I20260810 00:46:24.792886 10034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c413922be204a2182a4ae56c9d6976b [sys.catalog]: This master's current role is: LEADER
I20260810 00:46:24.795384 10041 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:46:24.800570 10041 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:46:24.809760 10041 catalog_manager.cc:1347] Generated new cluster ID: db1f9f96e1df4512ae2d89e6f8770eb5
I20260810 00:46:24.809859 10041 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:46:24.835403 10041 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:46:24.837173 10041 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:46:24.847503 10041 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c413922be204a2182a4ae56c9d6976b: Generated new TSK 0
I20260810 00:46:24.848474 10041 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 00:46:24.855191 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.1:0
--local_ip_for_outbound_sockets=127.28.6.1
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:36021
--builtin_ntp_servers=127.28.6.20:44313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 00:46:25.194294 10056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:46:25.194583 10056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:46:25.194733 10056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:46:25.201797 10056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:46:25.201916 10056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:46:25.202116 10056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.1
I20260810 00:46:25.211128 10056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:44313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:36021
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.1
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:46:25.212318 10056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:46:25.214133 10056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:46:25.225816 10068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:25.226787 10069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:25.227643 10071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:46:25.228394 10056 server_base.cc:1034] running on GCE node
I20260810 00:46:25.228920 10056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:46:25.229740 10056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:46:25.231001 10056 hybrid_clock.cc:648] HybridClock initialized: now 1786322785230937 us; error 51 us; skew 500 ppm
I20260810 00:46:25.231282 10056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 00:46:25.236333 10056 webserver.cc:460] Webserver started at http://127.28.6.1:41625/ using document root <none> and password file <none>
I20260810 00:46:25.237234 10056 fs_manager.cc:359] Metadata directory not provided
I20260810 00:46:25.237354 10056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:46:25.237664 10056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:46:25.240463 10056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/data/instance:
uuid: "86cc1deae41f4ac0a1201e11ab1785bf"
format_stamp: "Formatted at 2026-08-10 00:46:25 on dist-test-slave-1nl6"
server_key: "f450a8b86921fc9e952c866ae39f94d0"
server_key_iv: "189493212c5359351a27d6c681a54e74"
server_key_version: "encryptionkey@0"
I20260810 00:46:25.241252 10056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance:
uuid: "86cc1deae41f4ac0a1201e11ab1785bf"
format_stamp: "Formatted at 2026-08-10 00:46:25 on dist-test-slave-1nl6"
server_key: "f450a8b86921fc9e952c866ae39f94d0"
server_key_iv: "189493212c5359351a27d6c681a54e74"
server_key_version: "encryptionkey@0"
I20260810 00:46:25.246219 10056 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 00:46:25.249526 10078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:25.250766 10056 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 00:46:25.250975 10056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/wal
uuid: "86cc1deae41f4ac0a1201e11ab1785bf"
format_stamp: "Formatted at 2026-08-10 00:46:25 on dist-test-slave-1nl6"
server_key: "f450a8b86921fc9e952c866ae39f94d0"
server_key_iv: "189493212c5359351a27d6c681a54e74"
server_key_version: "encryptionkey@0"
I20260810 00:46:25.251186 10056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:46:25.281816 10056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:46:25.282620 10056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:46:25.282934 10056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:46:25.283855 10056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:46:25.285348 10056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:46:25.285458 10056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:25.285588 10056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:46:25.285681 10056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:25.319547 10056 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.1:38617
I20260810 00:46:25.319646 10250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.1:38617 every 8 connection(s)
I20260810 00:46:25.320779 10056 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
I20260810 00:46:25.322544 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 10056
I20260810 00:46:25.322741 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance
I20260810 00:46:25.323601 28696 external_mini_cluster.cc:1428] Setting key de7a8292430bd6b4bf06ac40c9b5befa
I20260810 00:46:25.326604 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.2:0
--local_ip_for_outbound_sockets=127.28.6.2
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:36021
--builtin_ntp_servers=127.28.6.20:44313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 00:46:25.335271 10257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:25.323262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:33283 (local address 127.28.6.62:36021)
0810 00:46:25.323780 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:25.323786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:25.325156 (+ 1370us) server_negotiation.cc:407] Connection header received
0810 00:46:25.327583 (+ 2427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:25.327587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:25.327650 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:25.327756 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:25.329193 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:25.329922 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:25.332424 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:25.332828 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:25.332928 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:25.334472 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:46:25.334476 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:46:25.334480 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:46:25.334498 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:46:25.334540 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:46:25.334552 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:46:25.334624 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:25.334880 (+ 256us) server_negotiation.cc:299] Negotiation successful
0810 00:46:25.335006 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":244,"threads_started":1}
I20260810 00:46:25.336650 10258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:25.323924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36021 (local address 127.28.6.1:33283)
0810 00:46:25.324971 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:25.325028 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:25.327297 (+ 2269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:25.328033 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:25.328047 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:25.328165 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:25.329005 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:25.329033 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:25.330072 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:25.330080 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:25.332259 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:25.332277 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:25.333053 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:25.333063 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:25.333384 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:25.334125 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 00:46:25.334131 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:25.334184 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 00:46:25.334280 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:46:25.334655 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:46:25.334669 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:46:25.334680 (+ 11us) client_negotiation.cc:769] Sending connection context
0810 00:46:25.334871 (+ 191us) client_negotiation.cc:240] Negotiation successful
0810 00:46:25.335302 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":359,"threads_started":1}
I20260810 00:46:25.338171 10251 heartbeater.cc:347] Connected to a master server at 127.28.6.62:36021
I20260810 00:46:25.338469 10251 heartbeater.cc:464] Registering TS with master...
I20260810 00:46:25.339277 10251 heartbeater.cc:511] Master 127.28.6.62:36021 requested a full tablet report, sending...
I20260810 00:46:25.341596 9974 ts_manager.cc:194] Registered new tserver with Master: 86cc1deae41f4ac0a1201e11ab1785bf (127.28.6.1:38617)
I20260810 00:46:25.343887 9974 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.6.1:33283
W20260810 00:46:25.673106 10259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:46:25.673357 10259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:46:25.673439 10259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:46:25.680440 10259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:46:25.680527 10259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:46:25.680624 10259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.2
I20260810 00:46:25.689301 10259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:44313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:36021
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.2
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:46:25.690459 10259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:46:25.692270 10259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:46:25.705724 10267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:25.705771 10264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:25.705701 10265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:46:25.706717 10259 server_base.cc:1034] running on GCE node
I20260810 00:46:25.707475 10259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:46:25.708493 10259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:46:25.709707 10259 hybrid_clock.cc:648] HybridClock initialized: now 1786322785709656 us; error 32 us; skew 500 ppm
I20260810 00:46:25.709991 10259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 00:46:25.712291 10259 webserver.cc:460] Webserver started at http://127.28.6.2:36825/ using document root <none> and password file <none>
I20260810 00:46:25.712924 10259 fs_manager.cc:359] Metadata directory not provided
I20260810 00:46:25.713014 10259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:46:25.713299 10259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:46:25.716161 10259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/data/instance:
uuid: "efc02f1e18764fb9b4cf104561dab851"
format_stamp: "Formatted at 2026-08-10 00:46:25 on dist-test-slave-1nl6"
server_key: "812aa20d118a8708cffcea3fa41e406f"
server_key_iv: "7311b21a05058321b11dcfb66c9618e3"
server_key_version: "encryptionkey@0"
I20260810 00:46:25.716935 10259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance:
uuid: "efc02f1e18764fb9b4cf104561dab851"
format_stamp: "Formatted at 2026-08-10 00:46:25 on dist-test-slave-1nl6"
server_key: "812aa20d118a8708cffcea3fa41e406f"
server_key_iv: "7311b21a05058321b11dcfb66c9618e3"
server_key_version: "encryptionkey@0"
I20260810 00:46:25.721975 10259 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 00:46:25.725265 10278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:25.726521 10259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 00:46:25.726655 10259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/wal
uuid: "efc02f1e18764fb9b4cf104561dab851"
format_stamp: "Formatted at 2026-08-10 00:46:25 on dist-test-slave-1nl6"
server_key: "812aa20d118a8708cffcea3fa41e406f"
server_key_iv: "7311b21a05058321b11dcfb66c9618e3"
server_key_version: "encryptionkey@0"
I20260810 00:46:25.726825 10259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:46:25.743836 10259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:46:25.744570 10259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:46:25.744796 10259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:46:25.745694 10259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:46:25.747102 10259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:46:25.747198 10259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:25.747283 10259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:46:25.747339 10259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:25.782471 10259 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.2:42349
I20260810 00:46:25.782596 10443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.2:42349 every 8 connection(s)
I20260810 00:46:25.784425 10259 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
I20260810 00:46:25.786504 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 10259
I20260810 00:46:25.786688 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance
I20260810 00:46:25.787334 28696 external_mini_cluster.cc:1428] Setting key ab0088273ba0ad22e5d6c0158e346a45
I20260810 00:46:25.790673 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.3:0
--local_ip_for_outbound_sockets=127.28.6.3
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:36021
--builtin_ntp_servers=127.28.6.20:44313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 00:46:25.799453 10257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:25.787261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:39827 (local address 127.28.6.62:36021)
0810 00:46:25.787488 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:25.787494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:25.789130 (+ 1636us) server_negotiation.cc:407] Connection header received
0810 00:46:25.791879 (+ 2749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:25.791885 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:25.791965 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:25.792059 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:25.793460 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:25.794239 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:25.796682 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:25.797051 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:25.797149 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:25.798706 (+ 1557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:46:25.798711 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:46:25.798714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:46:25.798732 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:46:25.798764 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:46:25.798776 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:46:25.798842 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:25.799139 (+ 297us) server_negotiation.cc:299] Negotiation successful
0810 00:46:25.799233 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 00:46:25.800807 10448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:25.787958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36021 (local address 127.28.6.2:39827)
0810 00:46:25.788957 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:25.789011 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:25.791556 (+ 2545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:25.792348 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:25.792363 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:25.792477 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:25.793267 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:25.793295 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:25.794396 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:25.794404 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:25.796535 (+ 2131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:25.796558 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:25.797275 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:25.797284 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:25.797617 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:25.798359 (+ 742us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 00:46:25.798365 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:25.798417 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 00:46:25.798510 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:46:25.798875 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:46:25.798889 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:46:25.798920 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 00:46:25.799113 (+ 193us) client_negotiation.cc:240] Negotiation successful
0810 00:46:25.799514 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":352,"threads_started":1}
I20260810 00:46:25.802153 10446 heartbeater.cc:347] Connected to a master server at 127.28.6.62:36021
I20260810 00:46:25.802467 10446 heartbeater.cc:464] Registering TS with master...
I20260810 00:46:25.803354 10446 heartbeater.cc:511] Master 127.28.6.62:36021 requested a full tablet report, sending...
I20260810 00:46:25.804787 9974 ts_manager.cc:194] Registered new tserver with Master: efc02f1e18764fb9b4cf104561dab851 (127.28.6.2:42349)
I20260810 00:46:25.805985 9974 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.6.2:39827
W20260810 00:46:26.133930 10450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:46:26.134204 10450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:46:26.134291 10450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:46:26.144845 10450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:46:26.144987 10450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:46:26.145170 10450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.3
I20260810 00:46:26.154397 10450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:44313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:36021
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.3
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:46:26.155827 10450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:46:26.157608 10450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:46:26.171301 10459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:26.171367 10462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:26.171253 10460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:46:26.172165 10450 server_base.cc:1034] running on GCE node
I20260810 00:46:26.172740 10450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:46:26.173573 10450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:46:26.174770 10450 hybrid_clock.cc:648] HybridClock initialized: now 1786322786174713 us; error 54 us; skew 500 ppm
I20260810 00:46:26.175092 10450 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 00:46:26.177171 10450 webserver.cc:460] Webserver started at http://127.28.6.3:35171/ using document root <none> and password file <none>
I20260810 00:46:26.177830 10450 fs_manager.cc:359] Metadata directory not provided
I20260810 00:46:26.177953 10450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:46:26.178275 10450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:46:26.181120 10450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/data/instance:
uuid: "e7eb6cfb271c43668a9c0c5ff37b8969"
format_stamp: "Formatted at 2026-08-10 00:46:26 on dist-test-slave-1nl6"
server_key: "3706a06f16a6fb63d9805e53dcf28ab9"
server_key_iv: "118f4a0f35a0a8527014b4f2fb0eeeaf"
server_key_version: "encryptionkey@0"
I20260810 00:46:26.181968 10450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance:
uuid: "e7eb6cfb271c43668a9c0c5ff37b8969"
format_stamp: "Formatted at 2026-08-10 00:46:26 on dist-test-slave-1nl6"
server_key: "3706a06f16a6fb63d9805e53dcf28ab9"
server_key_iv: "118f4a0f35a0a8527014b4f2fb0eeeaf"
server_key_version: "encryptionkey@0"
I20260810 00:46:26.187023 10450 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 00:46:26.190394 10472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:26.191771 10450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 00:46:26.191910 10450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/wal
uuid: "e7eb6cfb271c43668a9c0c5ff37b8969"
format_stamp: "Formatted at 2026-08-10 00:46:26 on dist-test-slave-1nl6"
server_key: "3706a06f16a6fb63d9805e53dcf28ab9"
server_key_iv: "118f4a0f35a0a8527014b4f2fb0eeeaf"
server_key_version: "encryptionkey@0"
I20260810 00:46:26.192083 10450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:46:26.204867 10450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:46:26.205569 10450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:46:26.205796 10450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:46:26.206660 10450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:46:26.208098 10450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:46:26.208186 10450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:26.208276 10450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:46:26.208330 10450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:26.240567 10450 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.3:32815
I20260810 00:46:26.240639 10644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.3:32815 every 8 connection(s)
I20260810 00:46:26.241813 10450 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
I20260810 00:46:26.249794 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 10450
I20260810 00:46:26.249974 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance
I20260810 00:46:26.250545 28696 external_mini_cluster.cc:1428] Setting key 1d2c8a453c8cd149f3aa7479f6d8a093
I20260810 00:46:26.254482 10257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:26.244113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:48413 (local address 127.28.6.62:36021)
0810 00:46:26.244341 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:26.244348 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:26.245647 (+ 1299us) server_negotiation.cc:407] Connection header received
0810 00:46:26.247397 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:26.247401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:26.247467 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:26.247554 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:26.248515 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:26.249231 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:26.251476 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:26.252105 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:26.252219 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:26.253818 (+ 1599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:46:26.253823 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:46:26.253826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:46:26.253843 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:46:26.253874 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:46:26.253885 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:46:26.253957 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:26.254180 (+ 223us) server_negotiation.cc:299] Negotiation successful
0810 00:46:26.254256 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 00:46:26.255650 10650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:26.244614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36021 (local address 127.28.6.3:48413)
0810 00:46:26.245491 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:26.245534 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:26.247160 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:26.247749 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:26.247756 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:26.247832 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:26.248369 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:26.248386 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:26.249362 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:26.249365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:26.251306 (+ 1941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:26.251332 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:26.252358 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:26.252367 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:26.252653 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:26.253440 (+ 787us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 00:46:26.253443 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:26.253494 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 00:46:26.253577 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:46:26.253977 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:46:26.253986 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:46:26.253991 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 00:46:26.254151 (+ 160us) client_negotiation.cc:240] Negotiation successful
0810 00:46:26.254466 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":282,"threads_started":1}
I20260810 00:46:26.256906 10645 heartbeater.cc:347] Connected to a master server at 127.28.6.62:36021
I20260810 00:46:26.257191 10645 heartbeater.cc:464] Registering TS with master...
I20260810 00:46:26.257880 10645 heartbeater.cc:511] Master 127.28.6.62:36021 requested a full tablet report, sending...
I20260810 00:46:26.259274 9974 ts_manager.cc:194] Registered new tserver with Master: e7eb6cfb271c43668a9c0c5ff37b8969 (127.28.6.3:32815)
I20260810 00:46:26.260025 9974 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.6.3:48413
I20260810 00:46:26.265681 28696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 00:46:26.271869 10257 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:26.270262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59852 (local address 127.28.6.62:36021)
0810 00:46:26.270450 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:26.270455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:26.271030 (+ 575us) server_negotiation.cc:407] Connection header received
0810 00:46:26.271159 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:26.271163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:26.271232 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:26.271311 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:26.271694 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59852: BlockingRecv error: recv got EOF from 127.0.0.1:59852 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 00:46:26.272607 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 10056
I20260810 00:46:26.279814 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:26.273426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46796 (local address 127.28.6.1:38617)
0810 00:46:26.273917 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:26.273930 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:26.273954 (+ 24us) server_negotiation.cc:407] Connection header received
0810 00:46:26.274007 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:26.274013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:26.274176 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:26.274290 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:26.274644 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:26.275665 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:26.276861 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:26.277392 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:26.277489 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:26.278418 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:46:26.278427 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:46:26.278438 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:46:26.279042 (+ 604us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:46:26.279085 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:46:26.279105 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:46:26.279210 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:26.279347 (+ 137us) server_negotiation.cc:299] Negotiation successful
0810 00:46:26.279447 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":241,"threads_started":1}
I20260810 00:46:26.280928 10244 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:46:26.361743 10251 heartbeater.cc:503] Master 127.28.6.62:36021 was elected leader, sending a full tablet report...
I20260810 00:46:26.363233 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 10056
W20260810 00:46:26.385938 9953 connection.cc:535] server connection from 127.28.6.1:33283 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 00:46:26.386303 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 10259
I20260810 00:46:26.394691 10664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:26.387278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49994 (local address 127.28.6.2:42349)
0810 00:46:26.387776 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:26.387784 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:26.387811 (+ 27us) server_negotiation.cc:407] Connection header received
0810 00:46:26.387881 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:26.387887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:26.388102 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:26.388255 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:26.388659 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:26.389889 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:26.391442 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:26.392039 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:26.392179 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:26.393195 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:46:26.393205 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:46:26.393219 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:46:26.393840 (+ 621us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:46:26.393888 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:46:26.393927 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:46:26.394021 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:26.394222 (+ 201us) server_negotiation.cc:299] Negotiation successful
0810 00:46:26.394375 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":235,"threads_started":1}
I20260810 00:46:26.396334 10438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:46:26.477552 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 10259
I20260810 00:46:26.504154 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 10450
I20260810 00:46:26.512524 10668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:26.505237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54244 (local address 127.28.6.3:32815)
0810 00:46:26.505940 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:26.505947 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:26.505979 (+ 32us) server_negotiation.cc:407] Connection header received
0810 00:46:26.506046 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:26.506052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:26.506264 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:26.506399 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:26.506835 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:26.507989 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:26.509301 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:26.509876 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:26.509981 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:26.511039 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:46:26.511048 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:46:26.511061 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:46:26.511644 (+ 583us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:46:26.511691 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:46:26.511728 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:46:26.511819 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:26.511967 (+ 148us) server_negotiation.cc:299] Negotiation successful
0810 00:46:26.512132 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":365,"threads_started":1}
I20260810 00:46:26.513756 10634 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:46:26.595826 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 10450
I20260810 00:46:26.623404 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 9933
I20260810 00:46:26.624651 10014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:46:26.729583 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 9933
2026-08-10T00:46:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2499 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-10T00:46:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:46:26Z Disabled control of system clock
I20260810 00:46:26.818199 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:43761
--webserver_interface=127.28.6.62
--webserver_port=0
--builtin_ntp_servers=127.28.6.20:40215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:43761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 00:46:27.145953 10674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:46:27.146225 10674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:46:27.146343 10674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:46:27.153568 10674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:46:27.153679 10674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:46:27.153767 10674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:46:27.153863 10674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:46:27.153939 10674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:46:27.162874 10674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:40215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:43761
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:43761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:46:27.164124 10674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:46:27.165877 10674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:46:27.177570 10683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:27.177678 10680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:27.177657 10685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:46:27.178465 10674 server_base.cc:1034] running on GCE node
I20260810 00:46:27.179291 10674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:46:27.180531 10674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:46:27.181736 10674 hybrid_clock.cc:648] HybridClock initialized: now 1786322787181694 us; error 37 us; skew 500 ppm
I20260810 00:46:27.182062 10674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 00:46:27.184157 10674 webserver.cc:460] Webserver started at http://127.28.6.62:34647/ using document root <none> and password file <none>
I20260810 00:46:27.184834 10674 fs_manager.cc:359] Metadata directory not provided
I20260810 00:46:27.184947 10674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:46:27.185274 10674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:46:27.188438 10674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "4a3ee0a8b38048dd9acef6903805dbed"
format_stamp: "Formatted at 2026-08-10 00:46:27 on dist-test-slave-1nl6"
server_key: "bf237d1a0ef154760245d468c9f286d8"
server_key_iv: "dc0ed1639409c456cb902701af6c60a4"
server_key_version: "encryptionkey@0"
I20260810 00:46:27.189220 10674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "4a3ee0a8b38048dd9acef6903805dbed"
format_stamp: "Formatted at 2026-08-10 00:46:27 on dist-test-slave-1nl6"
server_key: "bf237d1a0ef154760245d468c9f286d8"
server_key_iv: "dc0ed1639409c456cb902701af6c60a4"
server_key_version: "encryptionkey@0"
I20260810 00:46:27.194089 10674 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260810 00:46:27.197382 10694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:27.198562 10674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 00:46:27.198743 10674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "4a3ee0a8b38048dd9acef6903805dbed"
format_stamp: "Formatted at 2026-08-10 00:46:27 on dist-test-slave-1nl6"
server_key: "bf237d1a0ef154760245d468c9f286d8"
server_key_iv: "dc0ed1639409c456cb902701af6c60a4"
server_key_version: "encryptionkey@0"
I20260810 00:46:27.198988 10674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:46:27.204447 10674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:46:27.205163 10674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:46:27.205466 10674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:46:27.223340 10674 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:43761
I20260810 00:46:27.223412 10775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:43761 every 8 connection(s)
I20260810 00:46:27.224576 10674 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:46:27.228313 10776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:46:27.229300 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 10674
I20260810 00:46:27.229525 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:46:27.230194 28696 external_mini_cluster.cc:1428] Setting key 9509573024db7e5c286ffe42e3d8acf2
I20260810 00:46:27.238780 10779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:27.232127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60676 (local address 127.28.6.62:43761)
0810 00:46:27.233266 (+ 1139us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:27.233281 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:27.233336 (+ 55us) server_negotiation.cc:407] Connection header received
0810 00:46:27.234975 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:27.235006 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:27.235543 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:27.236001 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:27.236915 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:46:27.236923 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:46:27.236938 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:46:27.236954 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:46:27.237027 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:46:27.237047 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:46:27.237272 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:27.237395 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 00:46:27.237642 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":339,"threads_started":1}
I20260810 00:46:27.242724 10776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a3ee0a8b38048dd9acef6903805dbed: Bootstrap starting.
I20260810 00:46:27.245784 10776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a3ee0a8b38048dd9acef6903805dbed: Neither blocks nor log segments found. Creating new log.
I20260810 00:46:27.246863 10776 log.cc:824] T 00000000000000000000000000000000 P 4a3ee0a8b38048dd9acef6903805dbed: Log is configured to *not* fsync() on all Append() calls
I20260810 00:46:27.249305 10776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a3ee0a8b38048dd9acef6903805dbed: No bootstrap required, opened a new log
I20260810 00:46:27.256489 10776 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a3ee0a8b38048dd9acef6903805dbed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a3ee0a8b38048dd9acef6903805dbed" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 43761 } }
I20260810 00:46:27.256791 10776 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a3ee0a8b38048dd9acef6903805dbed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:46:27.256855 10776 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a3ee0a8b38048dd9acef6903805dbed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a3ee0a8b38048dd9acef6903805dbed, State: Initialized, Role: FOLLOWER
I20260810 00:46:27.257305 10776 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a3ee0a8b38048dd9acef6903805dbed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a3ee0a8b38048dd9acef6903805dbed" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 43761 } }
I20260810 00:46:27.257489 10776 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a3ee0a8b38048dd9acef6903805dbed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:46:27.257584 10776 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a3ee0a8b38048dd9acef6903805dbed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:46:27.257710 10776 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a3ee0a8b38048dd9acef6903805dbed [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:46:27.259806 10776 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a3ee0a8b38048dd9acef6903805dbed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a3ee0a8b38048dd9acef6903805dbed" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 43761 } }
I20260810 00:46:27.260385 10776 leader_election.cc:302] T 00000000000000000000000000000000 P 4a3ee0a8b38048dd9acef6903805dbed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a3ee0a8b38048dd9acef6903805dbed; no voters:
I20260810 00:46:27.261025 10776 leader_election.cc:288] T 00000000000000000000000000000000 P 4a3ee0a8b38048dd9acef6903805dbed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:46:27.261168 10782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a3ee0a8b38048dd9acef6903805dbed [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:46:27.261899 10782 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a3ee0a8b38048dd9acef6903805dbed [term 1 LEADER]: Becoming Leader. State: Replica: 4a3ee0a8b38048dd9acef6903805dbed, State: Running, Role: LEADER
I20260810 00:46:27.262431 10782 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a3ee0a8b38048dd9acef6903805dbed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a3ee0a8b38048dd9acef6903805dbed" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 43761 } }
I20260810 00:46:27.263095 10776 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a3ee0a8b38048dd9acef6903805dbed [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:46:27.265161 10785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a3ee0a8b38048dd9acef6903805dbed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a3ee0a8b38048dd9acef6903805dbed. Latest consensus state: current_term: 1 leader_uuid: "4a3ee0a8b38048dd9acef6903805dbed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a3ee0a8b38048dd9acef6903805dbed" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 43761 } } }
I20260810 00:46:27.265159 10784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a3ee0a8b38048dd9acef6903805dbed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a3ee0a8b38048dd9acef6903805dbed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a3ee0a8b38048dd9acef6903805dbed" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 43761 } } }
I20260810 00:46:27.265595 10785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a3ee0a8b38048dd9acef6903805dbed [sys.catalog]: This master's current role is: LEADER
I20260810 00:46:27.265745 10784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a3ee0a8b38048dd9acef6903805dbed [sys.catalog]: This master's current role is: LEADER
I20260810 00:46:27.270579 10788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:46:27.276216 10788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:46:27.284840 10788 catalog_manager.cc:1347] Generated new cluster ID: 7f2b101453aa4de9ab1af76fb3ac583a
I20260810 00:46:27.284955 10788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:46:27.316802 10788 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:46:27.317266 10788 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:46:27.329707 10788 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a3ee0a8b38048dd9acef6903805dbed: Generated new TSK 0
I20260810 00:46:27.330494 10788 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 00:46:27.343881 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.1:0
--local_ip_for_outbound_sockets=127.28.6.1
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:43761
--builtin_ntp_servers=127.28.6.20:40215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 00:46:27.698536 10808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:46:27.698820 10808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:46:27.698982 10808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:46:27.706315 10808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:46:27.706415 10808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:46:27.706521 10808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.1
I20260810 00:46:27.716339 10808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:40215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:43761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.1
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:46:27.717553 10808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:46:27.719374 10808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:46:27.732290 10819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:27.732345 10823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:27.732407 10817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:46:27.733230 10808 server_base.cc:1034] running on GCE node
I20260810 00:46:27.733785 10808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:46:27.734650 10808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:46:27.735872 10808 hybrid_clock.cc:648] HybridClock initialized: now 1786322787735804 us; error 65 us; skew 500 ppm
I20260810 00:46:27.736194 10808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 00:46:27.738451 10808 webserver.cc:460] Webserver started at http://127.28.6.1:43677/ using document root <none> and password file <none>
I20260810 00:46:27.739265 10808 fs_manager.cc:359] Metadata directory not provided
I20260810 00:46:27.739398 10808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:46:27.739723 10808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:46:27.744550 10808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/data/instance:
uuid: "a655dbf451e14c2ead6f86226855a6ab"
format_stamp: "Formatted at 2026-08-10 00:46:27 on dist-test-slave-1nl6"
server_key: "20bd0ffb40015f0627b6c1b8f28564c0"
server_key_iv: "b379d5a1b6a2924055043215243b5c80"
server_key_version: "encryptionkey@0"
I20260810 00:46:27.745348 10808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance:
uuid: "a655dbf451e14c2ead6f86226855a6ab"
format_stamp: "Formatted at 2026-08-10 00:46:27 on dist-test-slave-1nl6"
server_key: "20bd0ffb40015f0627b6c1b8f28564c0"
server_key_iv: "b379d5a1b6a2924055043215243b5c80"
server_key_version: "encryptionkey@0"
I20260810 00:46:27.750451 10808 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 00:46:27.753818 10835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:27.755103 10808 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 00:46:27.755267 10808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/wal
uuid: "a655dbf451e14c2ead6f86226855a6ab"
format_stamp: "Formatted at 2026-08-10 00:46:27 on dist-test-slave-1nl6"
server_key: "20bd0ffb40015f0627b6c1b8f28564c0"
server_key_iv: "b379d5a1b6a2924055043215243b5c80"
server_key_version: "encryptionkey@0"
I20260810 00:46:27.755475 10808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:46:27.761003 10808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:46:27.761737 10808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:46:27.762001 10808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:46:27.762859 10808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:46:27.764421 10808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:46:27.764510 10808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:27.764660 10808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:46:27.764753 10808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:27.797894 10808 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.1:41861
I20260810 00:46:27.797989 10992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.1:41861 every 8 connection(s)
I20260810 00:46:27.799274 10808 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
I20260810 00:46:27.804502 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 10808
I20260810 00:46:27.804724 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance
I20260810 00:46:27.805405 28696 external_mini_cluster.cc:1428] Setting key 0a9725d16a2b752c0d9ceb92d8af4eea
I20260810 00:46:27.808334 10997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:27.801777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:38973 (local address 127.28.6.62:43761)
0810 00:46:27.802228 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:27.802233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:27.803635 (+ 1402us) server_negotiation.cc:407] Connection header received
0810 00:46:27.805359 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:27.805365 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:27.805433 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:27.805573 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:27.807304 (+ 1731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:46:27.807309 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:46:27.807313 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:46:27.807329 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:46:27.807373 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:46:27.807416 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:46:27.807548 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:27.808062 (+ 514us) server_negotiation.cc:299] Negotiation successful
0810 00:46:27.808112 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":219,"threads_started":1}
I20260810 00:46:27.808856 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.2:0
--local_ip_for_outbound_sockets=127.28.6.2
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:43761
--builtin_ntp_servers=127.28.6.20:40215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 00:46:27.809579 10999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:27.802340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:43761 (local address 127.28.6.1:38973)
0810 00:46:27.803475 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:27.803519 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:27.805087 (+ 1568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:27.806045 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:27.806058 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:27.806185 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:27.806917 (+ 732us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 00:46:27.806921 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:27.806989 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 00:46:27.807096 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:46:27.807831 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:46:27.807842 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:46:27.807850 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 00:46:27.808146 (+ 296us) client_negotiation.cc:240] Negotiation successful
0810 00:46:27.808389 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":340,"threads_started":1}
I20260810 00:46:27.811021 10994 heartbeater.cc:347] Connected to a master server at 127.28.6.62:43761
I20260810 00:46:27.811344 10994 heartbeater.cc:464] Registering TS with master...
I20260810 00:46:27.812004 10994 heartbeater.cc:511] Master 127.28.6.62:43761 requested a full tablet report, sending...
I20260810 00:46:27.814040 10718 ts_manager.cc:194] Registered new tserver with Master: a655dbf451e14c2ead6f86226855a6ab (127.28.6.1:41861)
W20260810 00:46:28.150249 11000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:46:28.150710 11000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:46:28.150982 11000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:46:28.163897 11000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:46:28.164027 11000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:46:28.164242 11000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.2
I20260810 00:46:28.177466 11000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:40215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:43761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.2
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:46:28.178679 11000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:46:28.180488 11000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:46:28.193348 11008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:28.193534 11011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:28.193652 11009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:46:28.194856 11000 server_base.cc:1034] running on GCE node
I20260810 00:46:28.195542 11000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:46:28.196465 11000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:46:28.197697 11000 hybrid_clock.cc:648] HybridClock initialized: now 1786322788197648 us; error 54 us; skew 500 ppm
I20260810 00:46:28.198040 11000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 00:46:28.203426 11000 webserver.cc:460] Webserver started at http://127.28.6.2:34423/ using document root <none> and password file <none>
I20260810 00:46:28.204224 11000 fs_manager.cc:359] Metadata directory not provided
I20260810 00:46:28.204352 11000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:46:28.204720 11000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:46:28.207661 11000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/data/instance:
uuid: "3e211791935a461fba27ff59f5bbce30"
format_stamp: "Formatted at 2026-08-10 00:46:28 on dist-test-slave-1nl6"
server_key: "7e23b8d5c170bb0a11d3070073841443"
server_key_iv: "bbf345bab0fae3d39140bd18460d45c4"
server_key_version: "encryptionkey@0"
I20260810 00:46:28.208536 11000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance:
uuid: "3e211791935a461fba27ff59f5bbce30"
format_stamp: "Formatted at 2026-08-10 00:46:28 on dist-test-slave-1nl6"
server_key: "7e23b8d5c170bb0a11d3070073841443"
server_key_iv: "bbf345bab0fae3d39140bd18460d45c4"
server_key_version: "encryptionkey@0"
I20260810 00:46:28.213820 11000 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 00:46:28.217269 11019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:28.218559 11000 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 00:46:28.218742 11000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/wal
uuid: "3e211791935a461fba27ff59f5bbce30"
format_stamp: "Formatted at 2026-08-10 00:46:28 on dist-test-slave-1nl6"
server_key: "7e23b8d5c170bb0a11d3070073841443"
server_key_iv: "bbf345bab0fae3d39140bd18460d45c4"
server_key_version: "encryptionkey@0"
I20260810 00:46:28.218995 11000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:46:28.224699 11000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:46:28.225472 11000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:46:28.225755 11000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:46:28.226692 11000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:46:28.228312 11000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:46:28.228453 11000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:28.228575 11000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:46:28.228667 11000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:28.262470 11000 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.2:35923
I20260810 00:46:28.262554 11187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.2:35923 every 8 connection(s)
I20260810 00:46:28.264010 11000 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
I20260810 00:46:28.265316 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 11000
I20260810 00:46:28.265512 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance
I20260810 00:46:28.266074 28696 external_mini_cluster.cc:1428] Setting key 540992ffeb5a91203bf92d2a59ae3e69
I20260810 00:46:28.269259 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.3:0
--local_ip_for_outbound_sockets=127.28.6.3
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:43761
--builtin_ntp_servers=127.28.6.20:40215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 00:46:28.272912 10997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:28.266773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:53831 (local address 127.28.6.62:43761)
0810 00:46:28.267021 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:28.267029 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:28.268623 (+ 1594us) server_negotiation.cc:407] Connection header received
0810 00:46:28.270381 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:28.270386 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:28.270463 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:28.270582 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:28.272118 (+ 1536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:46:28.272124 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:46:28.272129 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:46:28.272147 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:46:28.272197 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:46:28.272209 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:46:28.272361 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:28.272558 (+ 197us) server_negotiation.cc:299] Negotiation successful
0810 00:46:28.272626 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 00:46:28.273768 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:28.267420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:43761 (local address 127.28.6.2:53831)
0810 00:46:28.268472 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:28.268515 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:28.270103 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:28.270809 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:28.270817 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:28.270954 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:28.271832 (+ 878us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 00:46:28.271834 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:28.271872 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 00:46:28.271954 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:46:28.272337 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:46:28.272348 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:46:28.272356 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 00:46:28.272644 (+ 288us) client_negotiation.cc:240] Negotiation successful
0810 00:46:28.272879 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":326,"threads_started":1}
I20260810 00:46:28.275143 11188 heartbeater.cc:347] Connected to a master server at 127.28.6.62:43761
I20260810 00:46:28.275458 11188 heartbeater.cc:464] Registering TS with master...
I20260810 00:46:28.276113 11188 heartbeater.cc:511] Master 127.28.6.62:43761 requested a full tablet report, sending...
I20260810 00:46:28.277825 10718 ts_manager.cc:194] Registered new tserver with Master: 3e211791935a461fba27ff59f5bbce30 (127.28.6.2:35923)
W20260810 00:46:28.626753 11192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:46:28.627045 11192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:46:28.627218 11192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:46:28.634231 11192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:46:28.634330 11192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:46:28.634480 11192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.3
I20260810 00:46:28.643184 11192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:40215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:43761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.3
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:46:28.644399 11192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:46:28.646155 11192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:46:28.659068 11199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:28.659116 11200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:28.659320 11203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:46:28.660399 11192 server_base.cc:1034] running on GCE node
I20260810 00:46:28.661046 11192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:46:28.662047 11192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:46:28.663281 11192 hybrid_clock.cc:648] HybridClock initialized: now 1786322788663233 us; error 50 us; skew 500 ppm
I20260810 00:46:28.663597 11192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 00:46:28.665891 11192 webserver.cc:460] Webserver started at http://127.28.6.3:38647/ using document root <none> and password file <none>
I20260810 00:46:28.666571 11192 fs_manager.cc:359] Metadata directory not provided
I20260810 00:46:28.666705 11192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:46:28.667104 11192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:46:28.669950 11192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/data/instance:
uuid: "921cac155f9e44a29d6097c3cc9a8790"
format_stamp: "Formatted at 2026-08-10 00:46:28 on dist-test-slave-1nl6"
server_key: "123870f0e91df0b745b2e1900537d695"
server_key_iv: "7ddb744f4abbe879d9befc4793191566"
server_key_version: "encryptionkey@0"
I20260810 00:46:28.670810 11192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance:
uuid: "921cac155f9e44a29d6097c3cc9a8790"
format_stamp: "Formatted at 2026-08-10 00:46:28 on dist-test-slave-1nl6"
server_key: "123870f0e91df0b745b2e1900537d695"
server_key_iv: "7ddb744f4abbe879d9befc4793191566"
server_key_version: "encryptionkey@0"
I20260810 00:46:28.676102 11192 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260810 00:46:28.680299 11212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:28.681691 11192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260810 00:46:28.681893 11192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/wal
uuid: "921cac155f9e44a29d6097c3cc9a8790"
format_stamp: "Formatted at 2026-08-10 00:46:28 on dist-test-slave-1nl6"
server_key: "123870f0e91df0b745b2e1900537d695"
server_key_iv: "7ddb744f4abbe879d9befc4793191566"
server_key_version: "encryptionkey@0"
I20260810 00:46:28.682118 11192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:46:28.687840 11192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:46:28.688616 11192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:46:28.688895 11192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:46:28.689844 11192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:46:28.691254 11192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:46:28.691375 11192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:28.691504 11192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:46:28.691591 11192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:28.726114 11192 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.3:33573
I20260810 00:46:28.726133 11385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.3:33573 every 8 connection(s)
I20260810 00:46:28.727521 11192 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
I20260810 00:46:28.735785 10997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:28.730228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:50471 (local address 127.28.6.62:43761)
0810 00:46:28.730446 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:28.730452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:28.731877 (+ 1425us) server_negotiation.cc:407] Connection header received
0810 00:46:28.733518 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:28.733522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:28.733602 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:28.733692 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:28.735166 (+ 1474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:46:28.735171 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:46:28.735174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:46:28.735189 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:46:28.735221 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:46:28.735232 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:46:28.735327 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:28.735520 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 00:46:28.735569 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 00:46:28.736438 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 11192
I20260810 00:46:28.736639 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance
I20260810 00:46:28.736847 11388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:28.730729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:43761 (local address 127.28.6.3:50471)
0810 00:46:28.731736 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:28.731771 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:28.733270 (+ 1499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:28.733979 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:28.733991 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:28.734096 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:28.734776 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 00:46:28.734778 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:28.734816 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 00:46:28.734990 (+ 174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:46:28.735328 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:46:28.735337 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:46:28.735343 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 00:46:28.735578 (+ 235us) client_negotiation.cc:240] Negotiation successful
0810 00:46:28.735776 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":369,"threads_started":1}
I20260810 00:46:28.737195 28696 external_mini_cluster.cc:1428] Setting key 38125adac337da9d6f98cbba2f1dfcbf
I20260810 00:46:28.738334 11386 heartbeater.cc:347] Connected to a master server at 127.28.6.62:43761
I20260810 00:46:28.738713 11386 heartbeater.cc:464] Registering TS with master...
I20260810 00:46:28.739574 11386 heartbeater.cc:511] Master 127.28.6.62:43761 requested a full tablet report, sending...
I20260810 00:46:28.741098 10718 ts_manager.cc:194] Registered new tserver with Master: 921cac155f9e44a29d6097c3cc9a8790 (127.28.6.3:33573)
I20260810 00:46:28.752863 28696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 00:46:28.759045 10997 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:46:28.757407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38402 (local address 127.28.6.62:43761)
0810 00:46:28.757577 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:28.757582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:28.758178 (+ 596us) server_negotiation.cc:407] Connection header received
0810 00:46:28.758347 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:28.758351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:28.758409 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:28.758488 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:28.758761 (+ 273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38402: BlockingRecv error: recv got EOF from 127.0.0.1:38402 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 00:46:28.759891 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 10808
I20260810 00:46:28.763468 11396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:28.760801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43002 (local address 127.28.6.1:41861)
0810 00:46:28.761345 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:28.761360 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:28.761387 (+ 27us) server_negotiation.cc:407] Connection header received
0810 00:46:28.761446 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:28.761452 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:28.761634 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:28.761767 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:28.762668 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:46:28.762675 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:46:28.762686 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:46:28.762700 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:46:28.762749 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:46:28.762768 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:46:28.762880 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:28.763028 (+ 148us) server_negotiation.cc:299] Negotiation successful
0810 00:46:28.763132 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":266,"threads_started":1}
I20260810 00:46:28.765761 10986 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:46:28.845597 10994 heartbeater.cc:503] Master 127.28.6.62:43761 was elected leader, sending a full tablet report...
I20260810 00:46:28.847097 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 10808
W20260810 00:46:28.872068 10699 connection.cc:535] server connection from 127.28.6.1:38973 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 00:46:28.872476 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 11000
I20260810 00:46:28.876112 11399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:28.873433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33562 (local address 127.28.6.2:35923)
0810 00:46:28.873922 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:28.873929 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:28.873956 (+ 27us) server_negotiation.cc:407] Connection header received
0810 00:46:28.874038 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:28.874044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:28.874254 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:28.874387 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:28.875392 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:46:28.875399 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:46:28.875411 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:46:28.875431 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:46:28.875476 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:46:28.875491 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:46:28.875596 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:28.875676 (+ 80us) server_negotiation.cc:299] Negotiation successful
0810 00:46:28.875792 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":212,"threads_started":1}
I20260810 00:46:28.877357 11181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:46:28.957834 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 11000
I20260810 00:46:28.986630 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 11192
I20260810 00:46:28.990590 11402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:28.987859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47008 (local address 127.28.6.3:33573)
0810 00:46:28.988598 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:28.988605 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:28.988638 (+ 33us) server_negotiation.cc:407] Connection header received
0810 00:46:28.988703 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:28.988709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:28.988928 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:28.989052 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:28.989948 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:46:28.989954 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:46:28.989965 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:46:28.989981 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:46:28.990019 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:46:28.990033 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:46:28.990164 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:28.990239 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 00:46:28.990321 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":420,"threads_started":1}
I20260810 00:46:28.991971 11379 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:46:29.073401 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 11192
I20260810 00:46:29.102766 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 10674
I20260810 00:46:29.104065 10766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:46:29.205034 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 10674
2026-08-10T00:46:29Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2475 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:46:29 dist-test-slave-1nl6 krb5kdc[11409](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:46:29 dist-test-slave-1nl6 krb5kdc[11409](info): set up 2 sockets
Aug 10 00:46:29 dist-test-slave-1nl6 krb5kdc[11409](info): commencing operation
krb5kdc: starting...
W20260810 00:46:30.511361 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.256s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:46:30 dist-test-slave-1nl6 krb5kdc[11409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:46:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:46:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.6.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
I20260810 00:46:30.762229 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:35613
--webserver_interface=127.28.6.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:35161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:35613
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:46:31.088119 11433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:46:31.088397 11433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:46:31.088559 11433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:46:31.095623 11433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:46:31.095714 11433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:46:31.095824 11433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:46:31.095930 11433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:46:31.096011 11433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:46:31.105404 11433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:35161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:35613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:35613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:46:31.106652 11433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:46:31.107972 11433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:46:31.119621 11440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:31.119670 11444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:31.119692 11441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:46:31.120811 11433 server_base.cc:1034] running on GCE node
I20260810 00:46:31.121587 11433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:46:31.123076 11433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:46:31.124327 11433 hybrid_clock.cc:648] HybridClock initialized: now 1786322791124289 us; error 49 us; skew 500 ppm
I20260810 00:46:31.124667 11433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:46:31 dist-test-slave-1nl6 krb5kdc[11409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322791, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:46:31.133028 11433 init.cc:377] Logged in from keytab as kudu/127.28.6.62@KRBTEST.COM (short username kudu)
I20260810 00:46:31.134419 11433 webserver.cc:460] Webserver started at http://127.28.6.62:39081/ using document root <none> and password file <none>
I20260810 00:46:31.135098 11433 fs_manager.cc:359] Metadata directory not provided
I20260810 00:46:31.135208 11433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:46:31.135469 11433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:46:31.138389 11433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "edb6aaf746274758bc21b3da61ed15ff"
format_stamp: "Formatted at 2026-08-10 00:46:31 on dist-test-slave-1nl6"
server_key: "90ecf637fdd79d9c87b10213b9d08463"
server_key_iv: "db81cb6b07faeb73d0fd8ea329540bbe"
server_key_version: "encryptionkey@0"
I20260810 00:46:31.139266 11433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "edb6aaf746274758bc21b3da61ed15ff"
format_stamp: "Formatted at 2026-08-10 00:46:31 on dist-test-slave-1nl6"
server_key: "90ecf637fdd79d9c87b10213b9d08463"
server_key_iv: "db81cb6b07faeb73d0fd8ea329540bbe"
server_key_version: "encryptionkey@0"
I20260810 00:46:31.144203 11433 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 00:46:31.147361 11459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:31.148599 11433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 00:46:31.148790 11433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "edb6aaf746274758bc21b3da61ed15ff"
format_stamp: "Formatted at 2026-08-10 00:46:31 on dist-test-slave-1nl6"
server_key: "90ecf637fdd79d9c87b10213b9d08463"
server_key_iv: "db81cb6b07faeb73d0fd8ea329540bbe"
server_key_version: "encryptionkey@0"
I20260810 00:46:31.149005 11433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:46:31.165153 11433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:46:31.169282 11433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:46:31.169544 11433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:46:31.187995 11433 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:35613
I20260810 00:46:31.188038 11528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:35613 every 8 connection(s)
I20260810 00:46:31.189288 11433 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:46:31.192778 11531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:46:31.193970 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 11433
I20260810 00:46:31.194212 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:46:31.194875 28696 external_mini_cluster.cc:1428] Setting key bac6dc1dd7fdb7b6ad9b283993faae49
I20260810 00:46:31.207005 11531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edb6aaf746274758bc21b3da61ed15ff: Bootstrap starting.
Aug 10 00:46:31 dist-test-slave-1nl6 krb5kdc[11409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:46:31.209934 11531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edb6aaf746274758bc21b3da61ed15ff: Neither blocks nor log segments found. Creating new log.
I20260810 00:46:31.211359 11531 log.cc:824] T 00000000000000000000000000000000 P edb6aaf746274758bc21b3da61ed15ff: Log is configured to *not* fsync() on all Append() calls
I20260810 00:46:31.214795 11531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edb6aaf746274758bc21b3da61ed15ff: No bootstrap required, opened a new log
I20260810 00:46:31.218847 11534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:31.196957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47036 (local address 127.28.6.62:35613)
0810 00:46:31.197844 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:31.197859 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:31.197919 (+ 60us) server_negotiation.cc:407] Connection header received
0810 00:46:31.199644 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:31.199672 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:31.200158 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:31.200612 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:31.202008 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:31.204078 (+ 2070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:31.205386 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:31.206368 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:31.206555 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:31.210344 (+ 3789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:46:31.210357 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:46:31.210377 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:46:31.210432 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:46:31.214415 (+ 3983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:46:31.215173 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:46:31.215181 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:46:31.215189 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:46:31.215433 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:46:31.215798 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:46:31.215803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:46:31.215806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:46:31.216501 (+ 695us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:46:31.216804 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:31.217101 (+ 297us) server_negotiation.cc:299] Negotiation successful
0810 00:46:31.217548 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":362,"threads_started":1}
I20260810 00:46:31.223825 11531 raft_consensus.cc:348] T 00000000000000000000000000000000 P edb6aaf746274758bc21b3da61ed15ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edb6aaf746274758bc21b3da61ed15ff" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 35613 } }
I20260810 00:46:31.224308 11531 raft_consensus.cc:374] T 00000000000000000000000000000000 P edb6aaf746274758bc21b3da61ed15ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:46:31.224438 11531 raft_consensus.cc:729] T 00000000000000000000000000000000 P edb6aaf746274758bc21b3da61ed15ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edb6aaf746274758bc21b3da61ed15ff, State: Initialized, Role: FOLLOWER
I20260810 00:46:31.225054 11531 consensus_queue.cc:260] T 00000000000000000000000000000000 P edb6aaf746274758bc21b3da61ed15ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edb6aaf746274758bc21b3da61ed15ff" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 35613 } }
I20260810 00:46:31.225261 11531 raft_consensus.cc:388] T 00000000000000000000000000000000 P edb6aaf746274758bc21b3da61ed15ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:46:31.225358 11531 raft_consensus.cc:482] T 00000000000000000000000000000000 P edb6aaf746274758bc21b3da61ed15ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:46:31.225479 11531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edb6aaf746274758bc21b3da61ed15ff [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:46:31.227558 11531 raft_consensus.cc:504] T 00000000000000000000000000000000 P edb6aaf746274758bc21b3da61ed15ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edb6aaf746274758bc21b3da61ed15ff" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 35613 } }
I20260810 00:46:31.227972 11531 leader_election.cc:302] T 00000000000000000000000000000000 P edb6aaf746274758bc21b3da61ed15ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edb6aaf746274758bc21b3da61ed15ff; no voters:
I20260810 00:46:31.228500 11531 leader_election.cc:288] T 00000000000000000000000000000000 P edb6aaf746274758bc21b3da61ed15ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:46:31.228658 11538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edb6aaf746274758bc21b3da61ed15ff [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:46:31.230500 11531 sys_catalog.cc:564] T 00000000000000000000000000000000 P edb6aaf746274758bc21b3da61ed15ff [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:46:31.230822 11538 raft_consensus.cc:686] T 00000000000000000000000000000000 P edb6aaf746274758bc21b3da61ed15ff [term 1 LEADER]: Becoming Leader. State: Replica: edb6aaf746274758bc21b3da61ed15ff, State: Running, Role: LEADER
I20260810 00:46:31.231391 11538 consensus_queue.cc:237] T 00000000000000000000000000000000 P edb6aaf746274758bc21b3da61ed15ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edb6aaf746274758bc21b3da61ed15ff" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 35613 } }
I20260810 00:46:31.234514 11541 sys_catalog.cc:455] T 00000000000000000000000000000000 P edb6aaf746274758bc21b3da61ed15ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader edb6aaf746274758bc21b3da61ed15ff. Latest consensus state: current_term: 1 leader_uuid: "edb6aaf746274758bc21b3da61ed15ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edb6aaf746274758bc21b3da61ed15ff" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 35613 } } }
I20260810 00:46:31.234579 11539 sys_catalog.cc:455] T 00000000000000000000000000000000 P edb6aaf746274758bc21b3da61ed15ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "edb6aaf746274758bc21b3da61ed15ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edb6aaf746274758bc21b3da61ed15ff" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 35613 } } }
I20260810 00:46:31.234946 11539 sys_catalog.cc:458] T 00000000000000000000000000000000 P edb6aaf746274758bc21b3da61ed15ff [sys.catalog]: This master's current role is: LEADER
I20260810 00:46:31.234952 11541 sys_catalog.cc:458] T 00000000000000000000000000000000 P edb6aaf746274758bc21b3da61ed15ff [sys.catalog]: This master's current role is: LEADER
I20260810 00:46:31.235793 11551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:46:31.240716 11551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:46:31.249274 11551 catalog_manager.cc:1347] Generated new cluster ID: b63fe95d5d3d49f0b781499848af5f6e
I20260810 00:46:31.249380 11551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:46:31.268658 11551 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:46:31.270407 11551 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:46:31.281605 11551 catalog_manager.cc:5868] T 00000000000000000000000000000000 P edb6aaf746274758bc21b3da61ed15ff: Generated new TSK 0
I20260810 00:46:31.282588 11551 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.6.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab.
Entry for principal kudu/127.28.6.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab.
I20260810 00:46:31.348369 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.1:0
--local_ip_for_outbound_sockets=127.28.6.1
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:35613
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:35161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:46:31.682755 11567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:46:31.683111 11567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:46:31.683283 11567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:46:31.691617 11567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:46:31.691757 11567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:46:31.691948 11567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.1
I20260810 00:46:31.701400 11567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:35161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:35613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.1
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:46:31.702600 11567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:46:31.703962 11567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:46:31.716820 11574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:31.716899 11575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:31.716858 11579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:46:31.717548 11567 server_base.cc:1034] running on GCE node
I20260810 00:46:31.718196 11567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:46:31.719156 11567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:46:31.720366 11567 hybrid_clock.cc:648] HybridClock initialized: now 1786322791720307 us; error 50 us; skew 500 ppm
I20260810 00:46:31.720671 11567 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:46:31 dist-test-slave-1nl6 krb5kdc[11409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322791, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:46:31.723862 11567 init.cc:377] Logged in from keytab as kudu/127.28.6.1@KRBTEST.COM (short username kudu)
I20260810 00:46:31.725275 11567 webserver.cc:460] Webserver started at http://127.28.6.1:38279/ using document root <none> and password file <none>
I20260810 00:46:31.726015 11567 fs_manager.cc:359] Metadata directory not provided
I20260810 00:46:31.726130 11567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:46:31.726403 11567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:46:31.729323 11567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/data/instance:
uuid: "5e4600c3e3aa4e8ab098d64f2ea4d79a"
format_stamp: "Formatted at 2026-08-10 00:46:31 on dist-test-slave-1nl6"
server_key: "4a4c709c0b9d6989fcaf98896eab1cb2"
server_key_iv: "ed4223e0a7c31dc9283a0294b1071031"
server_key_version: "encryptionkey@0"
I20260810 00:46:31.730003 11567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance:
uuid: "5e4600c3e3aa4e8ab098d64f2ea4d79a"
format_stamp: "Formatted at 2026-08-10 00:46:31 on dist-test-slave-1nl6"
server_key: "4a4c709c0b9d6989fcaf98896eab1cb2"
server_key_iv: "ed4223e0a7c31dc9283a0294b1071031"
server_key_version: "encryptionkey@0"
I20260810 00:46:31.735379 11567 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260810 00:46:31.738679 11590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:31.740070 11567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 00:46:31.740258 11567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/wal
uuid: "5e4600c3e3aa4e8ab098d64f2ea4d79a"
format_stamp: "Formatted at 2026-08-10 00:46:31 on dist-test-slave-1nl6"
server_key: "4a4c709c0b9d6989fcaf98896eab1cb2"
server_key_iv: "ed4223e0a7c31dc9283a0294b1071031"
server_key_version: "encryptionkey@0"
I20260810 00:46:31.740459 11567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:46:31.766331 11567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:46:31.770551 11567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:46:31.770853 11567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:46:31.771812 11567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:46:31.773109 11567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:46:31.773185 11567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:31.773356 11567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:46:31.773418 11567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:31.814803 11567 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.1:41165
I20260810 00:46:31.814841 11753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.1:41165 every 8 connection(s)
I20260810 00:46:31.816120 11567 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
I20260810 00:46:31.825248 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 11567
I20260810 00:46:31.825466 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance
I20260810 00:46:31.826124 28696 external_mini_cluster.cc:1428] Setting key 60665ab621b743a3d685b2a344813698
Aug 10 00:46:31 dist-test-slave-1nl6 krb5kdc[11409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322791, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:46:31.836140 11756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:31.818715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:40023 (local address 127.28.6.62:35613)
0810 00:46:31.819243 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:31.819248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:31.820134 (+ 886us) server_negotiation.cc:407] Connection header received
0810 00:46:31.821925 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:31.821931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:31.822022 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:31.822148 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:31.823681 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:31.824407 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:31.826063 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:31.826519 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:31.826660 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:31.830221 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:46:31.830226 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:46:31.830230 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:46:31.830310 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:46:31.832961 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:46:31.833673 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:46:31.833678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:46:31.833680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:46:31.833817 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:46:31.834309 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:46:31.834320 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:46:31.834323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:46:31.834576 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:46:31.834741 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:31.835555 (+ 814us) server_negotiation.cc:299] Negotiation successful
0810 00:46:31.835777 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":310,"threads_started":1}
I20260810 00:46:31.837011 11757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:31.819217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:35613 (local address 127.28.6.1:40023)
0810 00:46:31.819960 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:31.820009 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:31.821639 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:31.822389 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:31.822397 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:31.822955 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:31.823513 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:31.823530 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:31.824569 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:31.824573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:31.825908 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:31.825925 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:31.826750 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:31.826759 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:31.827050 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:31.827715 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:31.827737 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:31.830015 (+ 2278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:46:31.833106 (+ 3091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:46:31.833114 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:46:31.833128 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:46:31.833538 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:46:31.833946 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:46:31.833951 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:46:31.833955 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:46:31.834184 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:46:31.834706 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:46:31.834715 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:46:31.835132 (+ 417us) client_negotiation.cc:769] Sending connection context
0810 00:46:31.835439 (+ 307us) client_negotiation.cc:240] Negotiation successful
0810 00:46:31.835918 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":296,"threads_started":1}
I20260810 00:46:31.838582 11754 heartbeater.cc:347] Connected to a master server at 127.28.6.62:35613
I20260810 00:46:31.838874 11754 heartbeater.cc:464] Registering TS with master...
I20260810 00:46:31.839790 11754 heartbeater.cc:511] Master 127.28.6.62:35613 requested a full tablet report, sending...
I20260810 00:46:31.841801 11480 ts_manager.cc:194] Registered new tserver with Master: 5e4600c3e3aa4e8ab098d64f2ea4d79a (127.28.6.1:41165)
I20260810 00:46:31.843290 11480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.1@KRBTEST.COM'} at 127.28.6.1:40023
WARNING: no policy specified for kudu/127.28.6.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab.
Entry for principal kudu/127.28.6.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab.
I20260810 00:46:31.882223 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.2:0
--local_ip_for_outbound_sockets=127.28.6.2
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:35613
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:35161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:46:32.226202 11761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:46:32.226454 11761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:46:32.226653 11761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:46:32.234012 11761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:46:32.234144 11761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:46:32.234326 11761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.2
I20260810 00:46:32.243978 11761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:35161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:35613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.2
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:46:32.245215 11761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:46:32.246485 11761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:46:32.259806 11773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:32.259886 11775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:32.259886 11772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:46:32.260730 11761 server_base.cc:1034] running on GCE node
I20260810 00:46:32.261346 11761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:46:32.262224 11761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:46:32.263448 11761 hybrid_clock.cc:648] HybridClock initialized: now 1786322792263409 us; error 53 us; skew 500 ppm
I20260810 00:46:32.263782 11761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:46:32 dist-test-slave-1nl6 krb5kdc[11409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322792, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:46:32.266868 11761 init.cc:377] Logged in from keytab as kudu/127.28.6.2@KRBTEST.COM (short username kudu)
I20260810 00:46:32.268356 11761 webserver.cc:460] Webserver started at http://127.28.6.2:36593/ using document root <none> and password file <none>
I20260810 00:46:32.269022 11761 fs_manager.cc:359] Metadata directory not provided
I20260810 00:46:32.269174 11761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:46:32.269486 11761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:46:32.272433 11761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/data/instance:
uuid: "ff74fffaf8164271b2f812efd44757c0"
format_stamp: "Formatted at 2026-08-10 00:46:32 on dist-test-slave-1nl6"
server_key: "cfba1f0c8a3052e553a806350dae3a0b"
server_key_iv: "311035e3f53079cf970b9647c092e715"
server_key_version: "encryptionkey@0"
I20260810 00:46:32.273191 11761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance:
uuid: "ff74fffaf8164271b2f812efd44757c0"
format_stamp: "Formatted at 2026-08-10 00:46:32 on dist-test-slave-1nl6"
server_key: "cfba1f0c8a3052e553a806350dae3a0b"
server_key_iv: "311035e3f53079cf970b9647c092e715"
server_key_version: "encryptionkey@0"
I20260810 00:46:32.278211 11761 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 00:46:32.281618 11785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:32.282945 11761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 00:46:32.283084 11761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/wal
uuid: "ff74fffaf8164271b2f812efd44757c0"
format_stamp: "Formatted at 2026-08-10 00:46:32 on dist-test-slave-1nl6"
server_key: "cfba1f0c8a3052e553a806350dae3a0b"
server_key_iv: "311035e3f53079cf970b9647c092e715"
server_key_version: "encryptionkey@0"
I20260810 00:46:32.283247 11761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:46:32.306092 11761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:46:32.309813 11761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:46:32.310060 11761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:46:32.310916 11761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:46:32.312253 11761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:46:32.312330 11761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:32.312412 11761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:46:32.312456 11761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:32.347404 11761 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.2:34617
I20260810 00:46:32.347491 11960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.2:34617 every 8 connection(s)
I20260810 00:46:32.348706 11761 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
I20260810 00:46:32.354156 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 11761
I20260810 00:46:32.354379 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance
I20260810 00:46:32.354809 28696 external_mini_cluster.cc:1428] Setting key e5903526a01a78cf79822c1f27841021
Aug 10 00:46:32 dist-test-slave-1nl6 krb5kdc[11409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322792, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:46:32.370666 11965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:32.351193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:36651 (local address 127.28.6.62:35613)
0810 00:46:32.351709 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:32.351716 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:32.352844 (+ 1128us) server_negotiation.cc:407] Connection header received
0810 00:46:32.354954 (+ 2110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:32.354962 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:32.355065 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:32.355185 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:32.356868 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:32.357943 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:32.359541 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:32.360274 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:32.360430 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:32.364317 (+ 3887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:46:32.364322 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:46:32.364325 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:46:32.364357 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:46:32.367031 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:46:32.368000 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:46:32.368004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:46:32.368006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:46:32.368127 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:46:32.368640 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:46:32.368643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:46:32.368645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:46:32.368886 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:46:32.368992 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:32.369909 (+ 917us) server_negotiation.cc:299] Negotiation successful
0810 00:46:32.370153 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":227,"threads_started":1}
I20260810 00:46:32.371263 11966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:32.351819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:35613 (local address 127.28.6.2:36651)
0810 00:46:32.352658 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:32.352696 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:32.354648 (+ 1952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:32.355405 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:32.355413 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:32.356046 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:32.356662 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:32.356687 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:32.358110 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:32.358114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:32.359397 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:32.359409 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:32.360491 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:32.360496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:32.360688 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:32.361362 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:32.361384 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:32.364081 (+ 2697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:46:32.367167 (+ 3086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:46:32.367179 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:46:32.367196 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:46:32.367874 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:46:32.368243 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:46:32.368254 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:46:32.368260 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:46:32.368523 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:46:32.369019 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:46:32.369029 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:46:32.369454 (+ 425us) client_negotiation.cc:769] Sending connection context
0810 00:46:32.369786 (+ 332us) client_negotiation.cc:240] Negotiation successful
0810 00:46:32.370160 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":284,"threads_started":1}
I20260810 00:46:32.372722 11961 heartbeater.cc:347] Connected to a master server at 127.28.6.62:35613
I20260810 00:46:32.373086 11961 heartbeater.cc:464] Registering TS with master...
I20260810 00:46:32.374032 11961 heartbeater.cc:511] Master 127.28.6.62:35613 requested a full tablet report, sending...
I20260810 00:46:32.375715 11480 ts_manager.cc:194] Registered new tserver with Master: ff74fffaf8164271b2f812efd44757c0 (127.28.6.2:34617)
I20260810 00:46:32.376508 11480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.2@KRBTEST.COM'} at 127.28.6.2:36651
WARNING: no policy specified for kudu/127.28.6.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab.
Entry for principal kudu/127.28.6.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab.
I20260810 00:46:32.415762 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.3:0
--local_ip_for_outbound_sockets=127.28.6.3
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:35613
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:35161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:46:32.757428 11970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:46:32.757766 11970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:46:32.757942 11970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:46:32.765322 11970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:46:32.765477 11970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:46:32.765650 11970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.3
I20260810 00:46:32.775522 11970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:35161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:35613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.3
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:46:32.776866 11970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:46:32.778254 11970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:46:32.791941 11979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:32.791896 11983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:46:32.791896 11989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:46:32.792984 11970 server_base.cc:1034] running on GCE node
I20260810 00:46:32.793571 11970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:46:32.794484 11970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:46:32.795684 11970 hybrid_clock.cc:648] HybridClock initialized: now 1786322792795645 us; error 25 us; skew 500 ppm
I20260810 00:46:32.796008 11970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:46:32 dist-test-slave-1nl6 krb5kdc[11409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322792, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:46:32.799108 11970 init.cc:377] Logged in from keytab as kudu/127.28.6.3@KRBTEST.COM (short username kudu)
I20260810 00:46:32.800487 11970 webserver.cc:460] Webserver started at http://127.28.6.3:41305/ using document root <none> and password file <none>
I20260810 00:46:32.801162 11970 fs_manager.cc:359] Metadata directory not provided
I20260810 00:46:32.801319 11970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:46:32.801625 11970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:46:32.804646 11970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/data/instance:
uuid: "738aa35d4ee54069a67ee4e8de60c7e2"
format_stamp: "Formatted at 2026-08-10 00:46:32 on dist-test-slave-1nl6"
server_key: "2114330bc8102a7b12e214a775e6cb3f"
server_key_iv: "70c213d8cafea32ed61e6310bf035fdc"
server_key_version: "encryptionkey@0"
I20260810 00:46:32.805488 11970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance:
uuid: "738aa35d4ee54069a67ee4e8de60c7e2"
format_stamp: "Formatted at 2026-08-10 00:46:32 on dist-test-slave-1nl6"
server_key: "2114330bc8102a7b12e214a775e6cb3f"
server_key_iv: "70c213d8cafea32ed61e6310bf035fdc"
server_key_version: "encryptionkey@0"
I20260810 00:46:32.810694 11970 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260810 00:46:32.814057 12002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:32.815670 11970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 00:46:32.815850 11970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/wal
uuid: "738aa35d4ee54069a67ee4e8de60c7e2"
format_stamp: "Formatted at 2026-08-10 00:46:32 on dist-test-slave-1nl6"
server_key: "2114330bc8102a7b12e214a775e6cb3f"
server_key_iv: "70c213d8cafea32ed61e6310bf035fdc"
server_key_version: "encryptionkey@0"
I20260810 00:46:32.816059 11970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:46:32.846158 11754 heartbeater.cc:503] Master 127.28.6.62:35613 was elected leader, sending a full tablet report...
I20260810 00:46:32.849294 11970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:46:32.854031 11970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:46:32.854290 11970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:46:32.855445 11970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:46:32.856765 11970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:46:32.856849 11970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:32.856950 11970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:46:32.856997 11970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:46:32.891510 11970 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.3:45887
I20260810 00:46:32.891561 12182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.3:45887 every 8 connection(s)
I20260810 00:46:32.892865 11970 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
I20260810 00:46:32.900468 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 11970
I20260810 00:46:32.900666 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance
I20260810 00:46:32.901099 28696 external_mini_cluster.cc:1428] Setting key 0b3e1921e23a005138c83e8d5fcce115
Aug 10 00:46:32 dist-test-slave-1nl6 krb5kdc[11409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322792, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:46:32.912880 12188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:32.895471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:35411 (local address 127.28.6.62:35613)
0810 00:46:32.895964 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:32.895969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:32.896787 (+ 818us) server_negotiation.cc:407] Connection header received
0810 00:46:32.898635 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:32.898641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:32.898733 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:32.898851 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:32.900403 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:32.901362 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:32.903499 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:32.903925 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:32.904070 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:32.907645 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:46:32.907651 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:46:32.907654 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:46:32.907688 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:46:32.910139 (+ 2451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:46:32.910841 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:46:32.910845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:46:32.910847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:46:32.911022 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:46:32.911408 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:46:32.911412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:46:32.911414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:46:32.911668 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:46:32.911783 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:32.912434 (+ 651us) server_negotiation.cc:299] Negotiation successful
0810 00:46:32.912601 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":252,"threads_started":1}
I20260810 00:46:32.913812 12189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:32.895919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:35613 (local address 127.28.6.3:35411)
0810 00:46:32.896619 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:32.896665 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:32.898329 (+ 1664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:32.899122 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:32.899129 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:32.899639 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:46:32.900226 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:32.900246 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:32.901530 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:32.901535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:32.903359 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:32.903377 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:32.904205 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:32.904215 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:32.904507 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:32.905200 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:46:32.905222 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:46:32.907455 (+ 2233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:46:32.910278 (+ 2823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:46:32.910285 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:46:32.910297 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:46:32.910729 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:46:32.911119 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:46:32.911122 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:46:32.911125 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:46:32.911300 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:46:32.911775 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:46:32.911781 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:46:32.912092 (+ 311us) client_negotiation.cc:769] Sending connection context
0810 00:46:32.912331 (+ 239us) client_negotiation.cc:240] Negotiation successful
0810 00:46:32.912694 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":300,"threads_started":1}
I20260810 00:46:32.915304 12186 heartbeater.cc:347] Connected to a master server at 127.28.6.62:35613
I20260810 00:46:32.915653 12186 heartbeater.cc:464] Registering TS with master...
I20260810 00:46:32.916504 12186 heartbeater.cc:511] Master 127.28.6.62:35613 requested a full tablet report, sending...
I20260810 00:46:32.917994 11480 ts_manager.cc:194] Registered new tserver with Master: 738aa35d4ee54069a67ee4e8de60c7e2 (127.28.6.3:45887)
I20260810 00:46:32.918812 11480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.3@KRBTEST.COM'} at 127.28.6.3:35411
I20260810 00:46:32.928884 28696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 00:46:32.945487 12188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:32.933362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47038 (local address 127.28.6.62:35613)
0810 00:46:32.933535 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:32.933539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:32.934058 (+ 519us) server_negotiation.cc:407] Connection header received
0810 00:46:32.934208 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:32.934213 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:32.934292 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:32.934399 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:32.935790 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:32.936567 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:32.937722 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:32.938106 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:32.938195 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:32.940349 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:46:32.940354 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:46:32.940358 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:46:32.940387 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:46:32.942778 (+ 2391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:46:32.943459 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:46:32.943467 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:46:32.943475 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:46:32.943594 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:46:32.943979 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:46:32.943990 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:46:32.943993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:46:32.944353 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:46:32.944635 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:32.944972 (+ 337us) server_negotiation.cc:299] Negotiation successful
0810 00:46:32.945207 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 00:46:32.949215 11480 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 00:46:32.956009 11480 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 00:46:32.992147 12204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:32.967283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:34617 (local address 127.0.0.1:37582)
0810 00:46:32.980797 (+ 13514us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:32.980844 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:32.981201 (+ 357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:32.982662 (+ 1461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:32.982673 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:32.982716 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:46:32.983249 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:32.983266 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:32.985706 (+ 2440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:32.985711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:32.988720 (+ 3009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:32.988739 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:32.990934 (+ 2195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:32.990943 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:32.991214 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:32.991285 (+ 71us) client_negotiation.cc:769] Sending connection context
0810 00:46:32.991408 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 00:46:32.991632 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13194,"mutex_wait_us":28,"thread_start_us":13011,"threads_started":1}
I20260810 00:46:32.992301 12205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:32.967548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37582 (local address 127.28.6.2:34617)
0810 00:46:32.979879 (+ 12331us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:32.979893 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:32.980996 (+ 1103us) server_negotiation.cc:407] Connection header received
0810 00:46:32.981662 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:32.981676 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:32.982022 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:32.982225 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:46:32.983522 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:32.985499 (+ 1977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:32.988899 (+ 3400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:32.990378 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:32.990513 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:32.990753 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:32.991752 (+ 999us) server_negotiation.cc:299] Negotiation successful
0810 00:46:32.991942 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12093,"thread_start_us":12355,"threads_started":1}
I20260810 00:46:32.995687 12210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:32.967656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:45887 (local address 127.0.0.1:51438)
0810 00:46:32.984881 (+ 17225us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:32.984904 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:32.985046 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:32.987679 (+ 2633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:32.987684 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:32.987711 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:46:32.988192 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:32.988205 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:32.990256 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:32.990263 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:32.992682 (+ 2419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:32.992699 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:32.994718 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:32.994729 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:32.995007 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:32.995067 (+ 60us) client_negotiation.cc:769] Sending connection context
0810 00:46:32.995165 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 00:46:32.995327 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17070,"mutex_wait_us":37,"thread_start_us":19705,"threads_started":1}
I20260810 00:46:32.995867 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:32.968726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53056 (local address 127.28.6.1:41165)
0810 00:46:32.987001 (+ 18275us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:32.987009 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:32.987977 (+ 968us) server_negotiation.cc:407] Connection header received
0810 00:46:32.988094 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:32.988101 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:32.988334 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:32.988552 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:46:32.989169 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:32.990145 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:32.993183 (+ 3038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:32.994493 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:32.994629 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:32.994849 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:32.995381 (+ 532us) server_negotiation.cc:299] Negotiation successful
0810 00:46:32.995537 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18064,"thread_start_us":18272,"threads_started":1}
I20260810 00:46:32.996425 12212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:32.968362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51438 (local address 127.28.6.3:45887)
0810 00:46:32.986593 (+ 18231us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:32.986605 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:32.986646 (+ 41us) server_negotiation.cc:407] Connection header received
0810 00:46:32.986759 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:32.986767 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:32.987183 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:32.987394 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:46:32.988412 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:32.990103 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:32.992840 (+ 2737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:32.994031 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:32.994166 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:32.994390 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:32.995941 (+ 1551us) server_negotiation.cc:299] Negotiation successful
0810 00:46:32.996119 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18028,"thread_start_us":18533,"threads_started":1}
I20260810 00:46:32.996454 12206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:32.968139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:41165 (local address 127.0.0.1:53056)
0810 00:46:32.987693 (+ 19554us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:32.987711 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:32.987841 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:32.988532 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:32.988538 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:32.988564 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:46:32.989006 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:32.989015 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:32.990329 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:32.990334 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:32.993013 (+ 2679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:32.993029 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:32.994750 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:32.994758 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:32.994994 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:32.995060 (+ 66us) client_negotiation.cc:769] Sending connection context
0810 00:46:32.995164 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 00:46:32.996073 (+ 909us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19421,"mutex_wait_us":843,"thread_start_us":16413,"threads_started":1}
I20260810 00:46:32.997011 11858 tablet_service.cc:1467] Processing CreateTablet for tablet e9ec9455782042298f015f9e1067aae2 (DEFAULT_TABLE table=test-table [id=2e2624b2b6bf4c7b8001257cfafe8c68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:46:32.998382 11858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9ec9455782042298f015f9e1067aae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:46:33.000039 11662 tablet_service.cc:1467] Processing CreateTablet for tablet e9ec9455782042298f015f9e1067aae2 (DEFAULT_TABLE table=test-table [id=2e2624b2b6bf4c7b8001257cfafe8c68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:46:33.001551 11662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9ec9455782042298f015f9e1067aae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:46:33.002153 12083 tablet_service.cc:1467] Processing CreateTablet for tablet e9ec9455782042298f015f9e1067aae2 (DEFAULT_TABLE table=test-table [id=2e2624b2b6bf4c7b8001257cfafe8c68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:46:33.003549 12083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9ec9455782042298f015f9e1067aae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:46:33.012770 12214 tablet_bootstrap.cc:492] T e9ec9455782042298f015f9e1067aae2 P 5e4600c3e3aa4e8ab098d64f2ea4d79a: Bootstrap starting.
I20260810 00:46:33.015702 12217 tablet_bootstrap.cc:492] T e9ec9455782042298f015f9e1067aae2 P ff74fffaf8164271b2f812efd44757c0: Bootstrap starting.
I20260810 00:46:33.016472 12214 tablet_bootstrap.cc:654] T e9ec9455782042298f015f9e1067aae2 P 5e4600c3e3aa4e8ab098d64f2ea4d79a: Neither blocks nor log segments found. Creating new log.
I20260810 00:46:33.017885 12214 log.cc:824] T e9ec9455782042298f015f9e1067aae2 P 5e4600c3e3aa4e8ab098d64f2ea4d79a: Log is configured to *not* fsync() on all Append() calls
I20260810 00:46:33.019667 12218 tablet_bootstrap.cc:492] T e9ec9455782042298f015f9e1067aae2 P 738aa35d4ee54069a67ee4e8de60c7e2: Bootstrap starting.
I20260810 00:46:33.019804 12217 tablet_bootstrap.cc:654] T e9ec9455782042298f015f9e1067aae2 P ff74fffaf8164271b2f812efd44757c0: Neither blocks nor log segments found. Creating new log.
I20260810 00:46:33.021132 12217 log.cc:824] T e9ec9455782042298f015f9e1067aae2 P ff74fffaf8164271b2f812efd44757c0: Log is configured to *not* fsync() on all Append() calls
I20260810 00:46:33.023952 12217 tablet_bootstrap.cc:492] T e9ec9455782042298f015f9e1067aae2 P ff74fffaf8164271b2f812efd44757c0: No bootstrap required, opened a new log
I20260810 00:46:33.023967 12214 tablet_bootstrap.cc:492] T e9ec9455782042298f015f9e1067aae2 P 5e4600c3e3aa4e8ab098d64f2ea4d79a: No bootstrap required, opened a new log
I20260810 00:46:33.024236 12217 ts_tablet_manager.cc:1397] T e9ec9455782042298f015f9e1067aae2 P ff74fffaf8164271b2f812efd44757c0: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260810 00:46:33.024274 12214 ts_tablet_manager.cc:1397] T e9ec9455782042298f015f9e1067aae2 P 5e4600c3e3aa4e8ab098d64f2ea4d79a: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260810 00:46:33.024372 12218 tablet_bootstrap.cc:654] T e9ec9455782042298f015f9e1067aae2 P 738aa35d4ee54069a67ee4e8de60c7e2: Neither blocks nor log segments found. Creating new log.
I20260810 00:46:33.025699 12218 log.cc:824] T e9ec9455782042298f015f9e1067aae2 P 738aa35d4ee54069a67ee4e8de60c7e2: Log is configured to *not* fsync() on all Append() calls
I20260810 00:46:33.032534 12217 raft_consensus.cc:348] T e9ec9455782042298f015f9e1067aae2 P ff74fffaf8164271b2f812efd44757c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff74fffaf8164271b2f812efd44757c0" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34617 } } peers { permanent_uuid: "738aa35d4ee54069a67ee4e8de60c7e2" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45887 } } peers { permanent_uuid: "5e4600c3e3aa4e8ab098d64f2ea4d79a" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 41165 } }
I20260810 00:46:33.032918 12217 raft_consensus.cc:374] T e9ec9455782042298f015f9e1067aae2 P ff74fffaf8164271b2f812efd44757c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:46:33.033015 12217 raft_consensus.cc:729] T e9ec9455782042298f015f9e1067aae2 P ff74fffaf8164271b2f812efd44757c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff74fffaf8164271b2f812efd44757c0, State: Initialized, Role: FOLLOWER
I20260810 00:46:33.033087 12218 tablet_bootstrap.cc:492] T e9ec9455782042298f015f9e1067aae2 P 738aa35d4ee54069a67ee4e8de60c7e2: No bootstrap required, opened a new log
I20260810 00:46:33.033365 12218 ts_tablet_manager.cc:1397] T e9ec9455782042298f015f9e1067aae2 P 738aa35d4ee54069a67ee4e8de60c7e2: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260810 00:46:33.033656 12217 consensus_queue.cc:260] T e9ec9455782042298f015f9e1067aae2 P ff74fffaf8164271b2f812efd44757c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff74fffaf8164271b2f812efd44757c0" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34617 } } peers { permanent_uuid: "738aa35d4ee54069a67ee4e8de60c7e2" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45887 } } peers { permanent_uuid: "5e4600c3e3aa4e8ab098d64f2ea4d79a" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 41165 } }
I20260810 00:46:33.034807 11961 heartbeater.cc:503] Master 127.28.6.62:35613 was elected leader, sending a full tablet report...
I20260810 00:46:33.035800 12217 ts_tablet_manager.cc:1428] T e9ec9455782042298f015f9e1067aae2 P ff74fffaf8164271b2f812efd44757c0: Time spent starting tablet: real 0.011s user 0.013s sys 0.000s
I20260810 00:46:33.036092 12214 raft_consensus.cc:348] T e9ec9455782042298f015f9e1067aae2 P 5e4600c3e3aa4e8ab098d64f2ea4d79a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff74fffaf8164271b2f812efd44757c0" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34617 } } peers { permanent_uuid: "738aa35d4ee54069a67ee4e8de60c7e2" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45887 } } peers { permanent_uuid: "5e4600c3e3aa4e8ab098d64f2ea4d79a" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 41165 } }
I20260810 00:46:33.036545 12214 raft_consensus.cc:374] T e9ec9455782042298f015f9e1067aae2 P 5e4600c3e3aa4e8ab098d64f2ea4d79a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:46:33.036649 12214 raft_consensus.cc:729] T e9ec9455782042298f015f9e1067aae2 P 5e4600c3e3aa4e8ab098d64f2ea4d79a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e4600c3e3aa4e8ab098d64f2ea4d79a, State: Initialized, Role: FOLLOWER
I20260810 00:46:33.037253 12214 consensus_queue.cc:260] T e9ec9455782042298f015f9e1067aae2 P 5e4600c3e3aa4e8ab098d64f2ea4d79a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff74fffaf8164271b2f812efd44757c0" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34617 } } peers { permanent_uuid: "738aa35d4ee54069a67ee4e8de60c7e2" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45887 } } peers { permanent_uuid: "5e4600c3e3aa4e8ab098d64f2ea4d79a" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 41165 } }
I20260810 00:46:33.039031 12214 ts_tablet_manager.cc:1428] T e9ec9455782042298f015f9e1067aae2 P 5e4600c3e3aa4e8ab098d64f2ea4d79a: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260810 00:46:33.044415 12218 raft_consensus.cc:348] T e9ec9455782042298f015f9e1067aae2 P 738aa35d4ee54069a67ee4e8de60c7e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff74fffaf8164271b2f812efd44757c0" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34617 } } peers { permanent_uuid: "738aa35d4ee54069a67ee4e8de60c7e2" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45887 } } peers { permanent_uuid: "5e4600c3e3aa4e8ab098d64f2ea4d79a" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 41165 } }
I20260810 00:46:33.044752 12218 raft_consensus.cc:374] T e9ec9455782042298f015f9e1067aae2 P 738aa35d4ee54069a67ee4e8de60c7e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:46:33.044909 12218 raft_consensus.cc:729] T e9ec9455782042298f015f9e1067aae2 P 738aa35d4ee54069a67ee4e8de60c7e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 738aa35d4ee54069a67ee4e8de60c7e2, State: Initialized, Role: FOLLOWER
I20260810 00:46:33.045490 12218 consensus_queue.cc:260] T e9ec9455782042298f015f9e1067aae2 P 738aa35d4ee54069a67ee4e8de60c7e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff74fffaf8164271b2f812efd44757c0" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34617 } } peers { permanent_uuid: "738aa35d4ee54069a67ee4e8de60c7e2" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45887 } } peers { permanent_uuid: "5e4600c3e3aa4e8ab098d64f2ea4d79a" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 41165 } }
I20260810 00:46:33.046525 12186 heartbeater.cc:503] Master 127.28.6.62:35613 was elected leader, sending a full tablet report...
I20260810 00:46:33.047128 12218 ts_tablet_manager.cc:1428] T e9ec9455782042298f015f9e1067aae2 P 738aa35d4ee54069a67ee4e8de60c7e2: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
W20260810 00:46:33.068540 11755 tablet.cc:2369] T e9ec9455782042298f015f9e1067aae2 P 5e4600c3e3aa4e8ab098d64f2ea4d79a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 00:46:33.100791 11963 tablet.cc:2369] T e9ec9455782042298f015f9e1067aae2 P ff74fffaf8164271b2f812efd44757c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 00:46:33.144590 12187 tablet.cc:2369] T e9ec9455782042298f015f9e1067aae2 P 738aa35d4ee54069a67ee4e8de60c7e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:46:33.149940 12225 raft_consensus.cc:482] T e9ec9455782042298f015f9e1067aae2 P 5e4600c3e3aa4e8ab098d64f2ea4d79a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:46:33.150207 12225 raft_consensus.cc:504] T e9ec9455782042298f015f9e1067aae2 P 5e4600c3e3aa4e8ab098d64f2ea4d79a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff74fffaf8164271b2f812efd44757c0" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34617 } } peers { permanent_uuid: "738aa35d4ee54069a67ee4e8de60c7e2" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45887 } } peers { permanent_uuid: "5e4600c3e3aa4e8ab098d64f2ea4d79a" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 41165 } }
I20260810 00:46:33.151961 12225 leader_election.cc:288] T e9ec9455782042298f015f9e1067aae2 P 5e4600c3e3aa4e8ab098d64f2ea4d79a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff74fffaf8164271b2f812efd44757c0 (127.28.6.2:34617), 738aa35d4ee54069a67ee4e8de60c7e2 (127.28.6.3:45887)
I20260810 00:46:33.160871 12205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:33.152501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:44353 (local address 127.28.6.2:34617)
0810 00:46:33.152697 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:33.152705 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:33.153520 (+ 815us) server_negotiation.cc:407] Connection header received
0810 00:46:33.153775 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:33.153784 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:33.153878 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:33.153978 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:46:33.154597 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:33.155680 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:33.158395 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:33.159659 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:33.159799 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:33.159896 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:33.160476 (+ 580us) server_negotiation.cc:299] Negotiation successful
0810 00:46:33.160605 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 00:46:33.160869 12212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:33.152457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:50599 (local address 127.28.6.3:45887)
0810 00:46:33.152671 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:33.152677 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:33.153885 (+ 1208us) server_negotiation.cc:407] Connection header received
0810 00:46:33.154108 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:33.154115 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:33.154195 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:33.154272 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:46:33.154952 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:33.155937 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:33.158395 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:33.159659 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:33.159800 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:33.159900 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:33.160476 (+ 576us) server_negotiation.cc:299] Negotiation successful
0810 00:46:33.160605 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 00:46:33.161171 12231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:33.152559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:45887 (local address 127.28.6.1:50599)
0810 00:46:33.153772 (+ 1213us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:33.153786 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:33.153931 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:33.154368 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:33.154373 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:33.154404 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:46:33.154762 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:33.154771 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:33.156051 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:33.156056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:33.158211 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:33.158228 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:33.159969 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:33.159975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:33.160194 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:33.160306 (+ 112us) client_negotiation.cc:769] Sending connection context
0810 00:46:33.160426 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 00:46:33.160651 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1090,"mutex_wait_us":154,"thread_start_us":831,"threads_started":1}
I20260810 00:46:33.161449 12230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:33.152547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:34617 (local address 127.28.6.1:44353)
0810 00:46:33.153369 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0810 00:46:33.153395 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 00:46:33.153591 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:46:33.154026 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:46:33.154031 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:46:33.154065 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:46:33.154434 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:33.154453 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:33.155803 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:33.155807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:33.158211 (+ 2404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:46:33.158228 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:46:33.159954 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:46:33.159963 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:46:33.160209 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:33.160308 (+ 99us) client_negotiation.cc:769] Sending connection context
0810 00:46:33.160426 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 00:46:33.160651 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"mutex_wait_us":216,"thread_start_us":460,"threads_started":1}
I20260810 00:46:33.162320 12115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9ec9455782042298f015f9e1067aae2" candidate_uuid: "5e4600c3e3aa4e8ab098d64f2ea4d79a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "738aa35d4ee54069a67ee4e8de60c7e2" is_pre_election: true
I20260810 00:46:33.162552 11894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9ec9455782042298f015f9e1067aae2" candidate_uuid: "5e4600c3e3aa4e8ab098d64f2ea4d79a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff74fffaf8164271b2f812efd44757c0" is_pre_election: true
I20260810 00:46:33.162973 12115 raft_consensus.cc:2454] T e9ec9455782042298f015f9e1067aae2 P 738aa35d4ee54069a67ee4e8de60c7e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e4600c3e3aa4e8ab098d64f2ea4d79a in term 0.
I20260810 00:46:33.163121 11894 raft_consensus.cc:2454] T e9ec9455782042298f015f9e1067aae2 P ff74fffaf8164271b2f812efd44757c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e4600c3e3aa4e8ab098d64f2ea4d79a in term 0.
I20260810 00:46:33.163730 11594 leader_election.cc:302] T e9ec9455782042298f015f9e1067aae2 P 5e4600c3e3aa4e8ab098d64f2ea4d79a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e4600c3e3aa4e8ab098d64f2ea4d79a, 738aa35d4ee54069a67ee4e8de60c7e2; no voters:
I20260810 00:46:33.164134 12225 raft_consensus.cc:2781] T e9ec9455782042298f015f9e1067aae2 P 5e4600c3e3aa4e8ab098d64f2ea4d79a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 00:46:33.164289 12225 raft_consensus.cc:482] T e9ec9455782042298f015f9e1067aae2 P 5e4600c3e3aa4e8ab098d64f2ea4d79a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 00:46:33.164420 12225 raft_consensus.cc:3037] T e9ec9455782042298f015f9e1067aae2 P 5e4600c3e3aa4e8ab098d64f2ea4d79a [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:46:33.166590 12225 raft_consensus.cc:504] T e9ec9455782042298f015f9e1067aae2 P 5e4600c3e3aa4e8ab098d64f2ea4d79a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff74fffaf8164271b2f812efd44757c0" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34617 } } peers { permanent_uuid: "738aa35d4ee54069a67ee4e8de60c7e2" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45887 } } peers { permanent_uuid: "5e4600c3e3aa4e8ab098d64f2ea4d79a" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 41165 } }
I20260810 00:46:33.167253 12225 leader_election.cc:288] T e9ec9455782042298f015f9e1067aae2 P 5e4600c3e3aa4e8ab098d64f2ea4d79a [CANDIDATE]: Term 1 election: Requested vote from peers ff74fffaf8164271b2f812efd44757c0 (127.28.6.2:34617), 738aa35d4ee54069a67ee4e8de60c7e2 (127.28.6.3:45887)
I20260810 00:46:33.167716 11894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9ec9455782042298f015f9e1067aae2" candidate_uuid: "5e4600c3e3aa4e8ab098d64f2ea4d79a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff74fffaf8164271b2f812efd44757c0"
I20260810 00:46:33.167755 12115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9ec9455782042298f015f9e1067aae2" candidate_uuid: "5e4600c3e3aa4e8ab098d64f2ea4d79a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "738aa35d4ee54069a67ee4e8de60c7e2"
I20260810 00:46:33.167910 11894 raft_consensus.cc:3037] T e9ec9455782042298f015f9e1067aae2 P ff74fffaf8164271b2f812efd44757c0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:46:33.167987 12115 raft_consensus.cc:3037] T e9ec9455782042298f015f9e1067aae2 P 738aa35d4ee54069a67ee4e8de60c7e2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:46:33.170127 11894 raft_consensus.cc:2454] T e9ec9455782042298f015f9e1067aae2 P ff74fffaf8164271b2f812efd44757c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e4600c3e3aa4e8ab098d64f2ea4d79a in term 1.
I20260810 00:46:33.170249 12115 raft_consensus.cc:2454] T e9ec9455782042298f015f9e1067aae2 P 738aa35d4ee54069a67ee4e8de60c7e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e4600c3e3aa4e8ab098d64f2ea4d79a in term 1.
I20260810 00:46:33.170575 11592 leader_election.cc:302] T e9ec9455782042298f015f9e1067aae2 P 5e4600c3e3aa4e8ab098d64f2ea4d79a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e4600c3e3aa4e8ab098d64f2ea4d79a, ff74fffaf8164271b2f812efd44757c0; no voters:
I20260810 00:46:33.170959 12225 raft_consensus.cc:2781] T e9ec9455782042298f015f9e1067aae2 P 5e4600c3e3aa4e8ab098d64f2ea4d79a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:46:33.171422 12225 raft_consensus.cc:686] T e9ec9455782042298f015f9e1067aae2 P 5e4600c3e3aa4e8ab098d64f2ea4d79a [term 1 LEADER]: Becoming Leader. State: Replica: 5e4600c3e3aa4e8ab098d64f2ea4d79a, State: Running, Role: LEADER
I20260810 00:46:33.171892 12225 consensus_queue.cc:237] T e9ec9455782042298f015f9e1067aae2 P 5e4600c3e3aa4e8ab098d64f2ea4d79a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff74fffaf8164271b2f812efd44757c0" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34617 } } peers { permanent_uuid: "738aa35d4ee54069a67ee4e8de60c7e2" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45887 } } peers { permanent_uuid: "5e4600c3e3aa4e8ab098d64f2ea4d79a" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 41165 } }
I20260810 00:46:33.176292 11479 catalog_manager.cc:5495] T e9ec9455782042298f015f9e1067aae2 P 5e4600c3e3aa4e8ab098d64f2ea4d79a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e4600c3e3aa4e8ab098d64f2ea4d79a (127.28.6.1). New cstate: current_term: 1 leader_uuid: "5e4600c3e3aa4e8ab098d64f2ea4d79a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff74fffaf8164271b2f812efd44757c0" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "738aa35d4ee54069a67ee4e8de60c7e2" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e4600c3e3aa4e8ab098d64f2ea4d79a" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 41165 } health_report { overall_health: HEALTHY } } }
I20260810 00:46:33.238095 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:33.232450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53060 (local address 127.28.6.1:41165)
0810 00:46:33.232661 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:33.232669 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:33.232690 (+ 21us) server_negotiation.cc:407] Connection header received
0810 00:46:33.232978 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:33.232982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:33.233063 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:33.233242 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:46:33.233626 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:33.234418 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:33.235885 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:33.236315 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:33.236456 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:33.236982 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:46:33.237493 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:46:33.237682 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:33.237733 (+ 51us) server_negotiation.cc:299] Negotiation successful
0810 00:46:33.237816 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 00:46:33.246449 12115 raft_consensus.cc:1261] T e9ec9455782042298f015f9e1067aae2 P 738aa35d4ee54069a67ee4e8de60c7e2 [term 1 FOLLOWER]: Refusing update from remote peer 5e4600c3e3aa4e8ab098d64f2ea4d79a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:46:33.246451 11894 raft_consensus.cc:1261] T e9ec9455782042298f015f9e1067aae2 P ff74fffaf8164271b2f812efd44757c0 [term 1 FOLLOWER]: Refusing update from remote peer 5e4600c3e3aa4e8ab098d64f2ea4d79a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:46:33.247473 12233 consensus_queue.cc:1035] T e9ec9455782042298f015f9e1067aae2 P 5e4600c3e3aa4e8ab098d64f2ea4d79a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff74fffaf8164271b2f812efd44757c0" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:46:33.247836 12225 consensus_queue.cc:1035] T e9ec9455782042298f015f9e1067aae2 P 5e4600c3e3aa4e8ab098d64f2ea4d79a [LEADER]: Connected to new peer: Peer: permanent_uuid: "738aa35d4ee54069a67ee4e8de60c7e2" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:46:33.259989 12236 mvcc.cc:205] Tried to move back new op lower bound from 7316778161126621184 to 7316778160835776512. Current Snapshot: MvccSnapshot[applied={T|T < 7316778161126621184}]
I20260810 00:46:33.264030 12237 mvcc.cc:205] Tried to move back new op lower bound from 7316778161126621184 to 7316778160835776512. Current Snapshot: MvccSnapshot[applied={T|T < 7316778161126621184}]
I20260810 00:46:33.274250 11479 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:46:33.274533 11479 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:46:33.277990 11479 catalog_manager.cc:5782] T 00000000000000000000000000000000 P edb6aaf746274758bc21b3da61ed15ff: Sending DeleteTablet for 3 replicas of tablet e9ec9455782042298f015f9e1067aae2
I20260810 00:46:33.279115 11858 tablet_service.cc:1514] Processing DeleteTablet for tablet e9ec9455782042298f015f9e1067aae2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:46:33 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:37582
I20260810 00:46:33.279263 11662 tablet_service.cc:1514] Processing DeleteTablet for tablet e9ec9455782042298f015f9e1067aae2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:46:33 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:53056
I20260810 00:46:33.279434 12083 tablet_service.cc:1514] Processing DeleteTablet for tablet e9ec9455782042298f015f9e1067aae2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:46:33 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:51438
I20260810 00:46:33.281666 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 11567
I20260810 00:46:33.289361 12256 tablet_replica.cc:330] T e9ec9455782042298f015f9e1067aae2 P 5e4600c3e3aa4e8ab098d64f2ea4d79a: stopping tablet replica
I20260810 00:46:33.289907 12256 raft_consensus.cc:2229] T e9ec9455782042298f015f9e1067aae2 P 5e4600c3e3aa4e8ab098d64f2ea4d79a [term 1 LEADER]: Raft consensus shutting down.
I20260810 00:46:33.290696 12256 raft_consensus.cc:2258] T e9ec9455782042298f015f9e1067aae2 P 5e4600c3e3aa4e8ab098d64f2ea4d79a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:46:33.295969 12256 ts_tablet_manager.cc:1905] T e9ec9455782042298f015f9e1067aae2 P 5e4600c3e3aa4e8ab098d64f2ea4d79a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:46:33.299410 12258 tablet_replica.cc:330] T e9ec9455782042298f015f9e1067aae2 P 738aa35d4ee54069a67ee4e8de60c7e2: stopping tablet replica
I20260810 00:46:33.299847 12258 raft_consensus.cc:2229] T e9ec9455782042298f015f9e1067aae2 P 738aa35d4ee54069a67ee4e8de60c7e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:46:33.300352 12258 raft_consensus.cc:2258] T e9ec9455782042298f015f9e1067aae2 P 738aa35d4ee54069a67ee4e8de60c7e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:46:33.301416 12255 tablet_replica.cc:330] T e9ec9455782042298f015f9e1067aae2 P ff74fffaf8164271b2f812efd44757c0: stopping tablet replica
I20260810 00:46:33.301815 12255 raft_consensus.cc:2229] T e9ec9455782042298f015f9e1067aae2 P ff74fffaf8164271b2f812efd44757c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:46:33.302310 12255 raft_consensus.cc:2258] T e9ec9455782042298f015f9e1067aae2 P ff74fffaf8164271b2f812efd44757c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:46:33.302573 12258 ts_tablet_manager.cc:1905] T e9ec9455782042298f015f9e1067aae2 P 738aa35d4ee54069a67ee4e8de60c7e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:46:33.305236 12255 ts_tablet_manager.cc:1905] T e9ec9455782042298f015f9e1067aae2 P ff74fffaf8164271b2f812efd44757c0: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 10 00:46:33 dist-test-slave-1nl6 krb5kdc[11409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.1@KRBTEST.COM
I20260810 00:46:33.312310 12255 ts_tablet_manager.cc:1918] T e9ec9455782042298f015f9e1067aae2 P ff74fffaf8164271b2f812efd44757c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:46:33.312487 12255 log.cc:1196] T e9ec9455782042298f015f9e1067aae2 P ff74fffaf8164271b2f812efd44757c0: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-1/wal/wals/e9ec9455782042298f015f9e1067aae2
I20260810 00:46:33.312733 12256 ts_tablet_manager.cc:1918] T e9ec9455782042298f015f9e1067aae2 P 5e4600c3e3aa4e8ab098d64f2ea4d79a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:46:33.312937 12256 log.cc:1196] T e9ec9455782042298f015f9e1067aae2 P 5e4600c3e3aa4e8ab098d64f2ea4d79a: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-0/wal/wals/e9ec9455782042298f015f9e1067aae2
I20260810 00:46:33.313071 12255 ts_tablet_manager.cc:1939] T e9ec9455782042298f015f9e1067aae2 P ff74fffaf8164271b2f812efd44757c0: Deleting consensus metadata
I20260810 00:46:33.313545 12256 ts_tablet_manager.cc:1939] T e9ec9455782042298f015f9e1067aae2 P 5e4600c3e3aa4e8ab098d64f2ea4d79a: Deleting consensus metadata
I20260810 00:46:33.315222 11461 catalog_manager.cc:4841] TS 5e4600c3e3aa4e8ab098d64f2ea4d79a (127.28.6.1:41165): tablet e9ec9455782042298f015f9e1067aae2 (table test-table [id=2e2624b2b6bf4c7b8001257cfafe8c68]) successfully deleted
I20260810 00:46:33.315721 11463 catalog_manager.cc:4841] TS ff74fffaf8164271b2f812efd44757c0 (127.28.6.2:34617): tablet e9ec9455782042298f015f9e1067aae2 (table test-table [id=2e2624b2b6bf4c7b8001257cfafe8c68]) successfully deleted
I20260810 00:46:33.316811 12258 ts_tablet_manager.cc:1918] T e9ec9455782042298f015f9e1067aae2 P 738aa35d4ee54069a67ee4e8de60c7e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:46:33.316988 12258 log.cc:1196] T e9ec9455782042298f015f9e1067aae2 P 738aa35d4ee54069a67ee4e8de60c7e2: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786322687226232-28696-0/minicluster-data/ts-2/wal/wals/e9ec9455782042298f015f9e1067aae2
I20260810 00:46:33.317533 12258 ts_tablet_manager.cc:1939] T e9ec9455782042298f015f9e1067aae2 P 738aa35d4ee54069a67ee4e8de60c7e2: Deleting consensus metadata
I20260810 00:46:33.318593 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:33.284358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53068 (local address 127.28.6.1:41165)
0810 00:46:33.284585 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:33.284595 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:33.295483 (+ 10888us) server_negotiation.cc:407] Connection header received
0810 00:46:33.295901 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:33.295907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:33.295993 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:33.296083 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:33.297871 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:33.299133 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:33.302710 (+ 3577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:33.303440 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:33.303576 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:33.308822 (+ 5246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:46:33.308837 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:46:33.308854 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:46:33.308901 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:46:33.313499 (+ 4598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:46:33.316195 (+ 2696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:46:33.316205 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:46:33.316209 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:46:33.316367 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:46:33.316779 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:46:33.316783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:46:33.316785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:46:33.317247 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:46:33.317591 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:33.317949 (+ 358us) server_negotiation.cc:299] Negotiation successful
0810 00:46:33.318198 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 00:46:33.319317 11465 catalog_manager.cc:4841] TS 738aa35d4ee54069a67ee4e8de60c7e2 (127.28.6.3:45887): tablet e9ec9455782042298f015f9e1067aae2 (table test-table [id=2e2624b2b6bf4c7b8001257cfafe8c68]) successfully deleted
I20260810 00:46:33.319499 11747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:46:33.418138 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 11567
I20260810 00:46:33.445982 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 11761
Aug 10 00:46:33 dist-test-slave-1nl6 krb5kdc[11409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.2@KRBTEST.COM
I20260810 00:46:33.460875 12205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:33.446733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37594 (local address 127.28.6.2:34617)
0810 00:46:33.446945 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:33.446951 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:33.446976 (+ 25us) server_negotiation.cc:407] Connection header received
0810 00:46:33.447142 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:33.447147 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:33.447229 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:33.447325 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:33.448496 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:33.449227 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:33.450436 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:33.450826 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:33.450997 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:33.455235 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:46:33.455247 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:46:33.455263 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:46:33.455295 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:46:33.458305 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:46:33.458993 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:46:33.459005 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:46:33.459011 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:46:33.459154 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:46:33.459507 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:46:33.459511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:46:33.459513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:46:33.459921 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:46:33.460144 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:33.460420 (+ 276us) server_negotiation.cc:299] Negotiation successful
0810 00:46:33.460590 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 00:46:33.461591 11955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:46:33.552117 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 11761
I20260810 00:46:33.578825 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 11970
Aug 10 00:46:33 dist-test-slave-1nl6 krb5kdc[11409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.3@KRBTEST.COM
I20260810 00:46:33.593691 12212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:46:33.579683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51450 (local address 127.28.6.3:45887)
0810 00:46:33.579864 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 00:46:33.579868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:46:33.579887 (+ 19us) server_negotiation.cc:407] Connection header received
0810 00:46:33.580045 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:46:33.580049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:46:33.580119 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:46:33.580218 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:46:33.581414 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:33.582253 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:33.583624 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:46:33.584000 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:46:33.584096 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:46:33.588415 (+ 4319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:46:33.588432 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:46:33.588450 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:46:33.588482 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:46:33.591424 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:46:33.592019 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:46:33.592027 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:46:33.592029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:46:33.592149 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:46:33.592487 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:46:33.592490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:46:33.592492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:46:33.592843 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:46:33.593068 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0810 00:46:33.593272 (+ 204us) server_negotiation.cc:299] Negotiation successful
0810 00:46:33.593424 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 00:46:33.594367 12177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:46:33.682268 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 11970
I20260810 00:46:33.710146 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 11433
I20260810 00:46:33.711357 11523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:46:33.820480 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 11433
2026-08-10T00:46:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4617 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:46:33 dist-test-slave-1nl6 krb5kdc[12269](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:46:33 dist-test-slave-1nl6 krb5kdc[12269](info): set up 2 sockets
Aug 10 00:46:33 dist-test-slave-1nl6 krb5kdc[12269](info): commencing operation
krb5kdc: starting...
W20260810 00:46:35.143543 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.271s user 0.000s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:46:35 dist-test-slave-1nl6 krb5kdc[12269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:46:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:46:35Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.6.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.6.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.6.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/krb5kdc/rangeradmin_127.28.6.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.6.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/krb5kdc/rangeradmin_127.28.6.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.6.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.6.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.6.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/krb5kdc/rangerlookup_127.28.6.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.6.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/krb5kdc/rangerlookup_127.28.6.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.6.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.6.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.6.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/krb5kdc/HTTP_127.28.6.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.6.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/krb5kdc/HTTP_127.28.6.20@KRBTEST.COM.keytab.
I20260810 00:46:35.529148 28696 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasksi0adg/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/postgres -l logfile start
2026-08-10 00:46:37.947 UTC [12309] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 00:46:37.947 UTC [12309] LOG: listening on IPv4 address "127.28.6.20", port 38727
2026-08-10 00:46:37.950 UTC [12309] LOG: listening on Unix socket "/tmp/.s.PGSQL.38727"
2026-08-10 00:46:37.972 UTC [12313] LOG: database system was shut down at 2026-08-10 00:46:36 UTC
2026-08-10 00:46:37.975 UTC [12309] LOG: database system is ready to accept connections
I20260810 00:46:39.169597 28696 mini_postgres.cc:96] Postgres bound to 38727
2026-08-10 00:46:39.193 UTC [12322] FATAL: database "slave" does not exist
127.28.6.20:38727 - accepting connections
I20260810 00:46:39.193961 28696 mini_ranger.cc:161] Starting Ranger...
I20260810 00:46:39.221818 28696 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 00:46:39.643339 28696 mini_ranger.cc:87] Created ranger Postgres database
I20260810 00:46:39.643494 28696 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ranger-admin
2026-08-10 00:46:39,973 [I] DB FLAVOR :POSTGRES
2026-08-10 00:46:39,974 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 00:46:39,974 [I] Checking connection..
2026-08-10 00:46:39,974 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:38727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 00:46:40,220 [I] Checking connection passed.
2026-08-10 00:46:40,220 [I] --------- Verifying version history table ---------
2026-08-10 00:46:40,220 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:38727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 00:46:40,448 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 00:46:40,448 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:38727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 00:46:40,690 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 00:46:40,690 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 00:46:40,690 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:38727/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 00:46:40.915 UTC [12422] WARNING: there is no transaction in progress
2026-08-10 00:46:40,923 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 00:46:40,924 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:38727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 00:46:41,160 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 00:46:41,161 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 00:46:41,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:38727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 00:46:41,421 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:38727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 00:46:41,660 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:38727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-1nl6.c.gcp-upstream.internal','N') ;"
2026-08-10 00:46:41,904 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:38727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 00:46:42,164 [I] Table x_portal_user does not exist in database ranger
2026-08-10 00:46:42,164 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:38727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 00:46:42,417 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 00:46:42,417 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:38727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 00:46:42,662 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:38727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 00:46:42,917 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 00:46:42,917 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:38727/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 00:46:43.813 UTC [12578] WARNING: there is no transaction in progress
2026-08-10 00:46:43.828 UTC [12578] WARNING: there is no transaction in progress
2026-08-10 00:46:43.843 UTC [12578] WARNING: there is no transaction in progress
2026-08-10 00:46:43.857 UTC [12578] WARNING: there is no transaction in progress
2026-08-10 00:46:43.872 UTC [12578] WARNING: there is no transaction in progress
2026-08-10 00:46:43.892 UTC [12578] WARNING: there is no transaction in progress
2026-08-10 00:46:43.945 UTC [12578] WARNING: there is no transaction in progress
2026-08-10 00:46:43.951 UTC [12578] WARNING: there is no transaction in progress
2026-08-10 00:46:43.965 UTC [12578] WARNING: there is no transaction in progress
2026-08-10 00:46:43.974 UTC [12578] WARNING: there is no transaction in progress
2026-08-10 00:46:43.984 UTC [12578] WARNING: there is no transaction in progress
2026-08-10 00:46:43.997 UTC [12578] WARNING: there is no transaction in progress
2026-08-10 00:46:44.006 UTC [12578] WARNING: there is no transaction in progress
2026-08-10 00:46:44.012 UTC [12578] WARNING: there is no transaction in progress
2026-08-10 00:46:44.019 UTC [12578] WARNING: there is no transaction in progress
2026-08-10 00:46:44.025 UTC [12578] WARNING: there is no transaction in progress
2026-08-10 00:46:44,604 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 00:46:44,604 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:38727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 00:46:44,841 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 00:46:44,842 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:38727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 00:46:45,100 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 00:46:45,100 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:38727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 00:46:45,363 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 00:46:45,363 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:38727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 00:46:45,589 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:38727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 00:46:45,846 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:38727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-1nl6.c.gcp-upstream.internal';"
2026-08-10 00:46:46,102 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 00:46:46,102 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:38727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 00:46:46,363 [I] DB_PATCHES have already been applied
I20260810 00:46:46.369678 28696 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ranger-admin:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasksi0adg/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 00:46:46.369765 28696 mini_ranger.cc:193] Using host: 127.28.6.20
I20260810 00:46:46.390439 28696 mini_ranger.cc:227] Ranger admin URL: http://127.28.6.20:53109
Aug 10, 2026 12:46:46 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 12:46:46 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 12:46:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 12:46:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 12:46:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 12:46:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 12:46:47 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 00:46:47.766119 28696 mini_ranger.cc:160] Time spent starting Ranger: real 8.572s user 0.000s sys 0.054s
Aug 10, 2026 12:47:07 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 12:47:07 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 12:47:07 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 12:47:08 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 12:47:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 12:47:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 12:47:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 12:47:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 12:47:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 12:47:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 12:47:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 12:47:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 12:47:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 12:47:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 12:47:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 12:47:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 12:47:08 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 12:47:08 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 00:47:09.457315 28696 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.6.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.6.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.6.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/krb5kdc/rangerkms_127.28.6.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.6.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/krb5kdc/rangerkms_127.28.6.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.6.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.6.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.6.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/krb5kdc/HTTP_127.28.6.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.6.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/krb5kdc/HTTP_127.28.6.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 00:47:09 dist-test-slave-1nl6 krb5kdc[12269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322829, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 00:47:09.647001 28696 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 00:47:09.671111 28696 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 00:47:10.443678 28696 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 00:47:10.443971 28696 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ranger-kms
I20260810 00:47:10.444125 28696 mini_ranger_kms.cc:227] Using postgres at 127.28.6.20:38727
2026-08-10 00:47:11,018 [I] DB FLAVOR :POSTGRES
2026-08-10 00:47:11,019 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 00:47:11,019 [I] Checking connection
2026-08-10 00:47:11,019 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:38727/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 00:47:11,277 [I] connection success
2026-08-10 00:47:11,277 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 00:47:11,277 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 00:47:11,277 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:38727/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 00:47:11,520 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 00:47:11,520 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 00:47:11,520 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 00:47:11,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:38727/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 00:47:11,770 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 00:47:12.261581 28696 mini_ranger_kms.cc:310] Created kms service
I20260810 00:47:12.476934 28696 mini_ranger_kms.cc:326] Created kudu user
I20260810 00:47:12.521600 28696 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 00:47:12 dist-test-slave-1nl6 krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322829, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.6.20@KRBTEST.COM
I20260810 00:47:12.939343 28696 mini_ranger_kms.cc:384] Added ranger policy
I20260810 00:47:12.940893 28696 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ranger-kms:/tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 00:47:12.940972 28696 mini_ranger_kms.cc:240] Using host: 127.28.6.20
I20260810 00:47:12.966141 28696 mini_ranger_kms.cc:276] Ranger KMS PID: 12848
I20260810 00:47:12.966367 28696 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.6.20:59437
Aug 10, 2026 12:47:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 12:47:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 12:47:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 12:47:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 12:47:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 12:47:14 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 00:47:15 dist-test-slave-1nl6 krb5kdc[12269](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786322835, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.6.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:47:15.766949 28696 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.120s user 0.157s sys 0.449s
I20260810 00:47:15.767308 28696 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 00:47:15.767526 28696 mini_ranger_kms.cc:401] 127.28.6.20:59437/kms/v1/keys
Aug 10 00:47:15 dist-test-slave-1nl6 krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322835, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.6.20@KRBTEST.COM for HTTP/127.28.6.20@KRBTEST.COM
Aug 10 00:47:16 dist-test-slave-1nl6 krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322835, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.6.20@KRBTEST.COM for HTTP/127.28.6.20@KRBTEST.COM
Aug 10 00:47:18 dist-test-slave-1nl6 krb5kdc[12269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.6.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
I20260810 00:47:18.872529 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:45435
--webserver_interface=127.28.6.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:42411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:45435
--ranger_config_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.6.20:59437/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:47:19.531760 12921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:47:19.532195 12921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:47:19.532426 12921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:47:19.542552 12921 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:47:19.542783 12921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:47:19.542984 12921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:47:19.543184 12921 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:47:19.543346 12921 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:47:19.556739 12921 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:42411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.6.20:59437/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:45435
--ranger_config_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:45435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:47:19.558233 12921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:47:19.559844 12921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:47:19.579031 12932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:47:19.579378 12933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:47:19.580245 12935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:47:19.582056 12921 server_base.cc:1034] running on GCE node
I20260810 00:47:19.583141 12921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:47:19.584653 12921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:47:19.585883 12921 hybrid_clock.cc:648] HybridClock initialized: now 1786322839585828 us; error 49 us; skew 500 ppm
I20260810 00:47:19.586252 12921 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:47:19 dist-test-slave-1nl6 krb5kdc[12269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322839, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:47:19.590634 12921 init.cc:377] Logged in from keytab as kudu/127.28.6.62@KRBTEST.COM (short username kudu)
I20260810 00:47:19.592882 12921 webserver.cc:460] Webserver started at http://127.28.6.62:38275/ using document root <none> and password file <none>
I20260810 00:47:19.593714 12921 fs_manager.cc:359] Metadata directory not provided
I20260810 00:47:19.593986 12921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:47:19.594347 12921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 00:47:19 dist-test-slave-1nl6 krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322839, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for HTTP/127.28.6.20@KRBTEST.COM
I20260810 00:47:19.666345 12921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "dff4912c85064f6784b00d6b6fb06f3f"
format_stamp: "Formatted at 2026-08-10 00:47:19 on dist-test-slave-1nl6"
server_key: "7c44261265c2db8645d24a6f20746b1e"
server_key_iv: "7b680b38e03f5e7e92a37e967c4bd908"
server_key_version: "kuduclusterkey@0"
I20260810 00:47:19.667683 12921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "dff4912c85064f6784b00d6b6fb06f3f"
format_stamp: "Formatted at 2026-08-10 00:47:19 on dist-test-slave-1nl6"
server_key: "7c44261265c2db8645d24a6f20746b1e"
server_key_iv: "7b680b38e03f5e7e92a37e967c4bd908"
server_key_version: "kuduclusterkey@0"
I20260810 00:47:19.677649 12921 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.005s
I20260810 00:47:19.701882 12944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:47:19.703940 12921 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.001s
I20260810 00:47:19.704233 12921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "dff4912c85064f6784b00d6b6fb06f3f"
format_stamp: "Formatted at 2026-08-10 00:47:19 on dist-test-slave-1nl6"
server_key: "7c44261265c2db8645d24a6f20746b1e"
server_key_iv: "7b680b38e03f5e7e92a37e967c4bd908"
server_key_version: "kuduclusterkey@0"
I20260810 00:47:19.704595 12921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:47:19.732445 12921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:47:19.740058 12921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:47:19.740559 12921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:47:19.787506 12921 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:45435
I20260810 00:47:19.787684 13026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:45435 every 8 connection(s)
I20260810 00:47:19.789474 12921 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:47:19.794858 13027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:47:19.798974 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 12921
I20260810 00:47:19.799263 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:47:19.800431 28696 external_mini_cluster.cc:1428] Setting key 566e0c384fe8f1ac6ff860450a5e4134
I20260810 00:47:19.814263 13027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dff4912c85064f6784b00d6b6fb06f3f: Bootstrap starting.
Aug 10 00:47:19 dist-test-slave-1nl6 krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:47:19.819962 13027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dff4912c85064f6784b00d6b6fb06f3f: Neither blocks nor log segments found. Creating new log.
I20260810 00:47:19.821987 13027 log.cc:824] T 00000000000000000000000000000000 P dff4912c85064f6784b00d6b6fb06f3f: Log is configured to *not* fsync() on all Append() calls
I20260810 00:47:19.827565 13027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dff4912c85064f6784b00d6b6fb06f3f: No bootstrap required, opened a new log
I20260810 00:47:19.830832 13031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:47:19.803452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52858 (local address 127.28.6.62:45435)
0810 00:47:19.805307 (+ 1855us) server_negotiation.cc:206] Beginning negotiation
0810 00:47:19.805320 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 00:47:19.805379 (+ 59us) server_negotiation.cc:407] Connection header received
0810 00:47:19.808240 (+ 2861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:47:19.808268 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:47:19.808666 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:47:19.809068 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:47:19.810458 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:47:19.812782 (+ 2324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:47:19.814123 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:47:19.815166 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:47:19.815370 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:47:19.819464 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:47:19.819473 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:47:19.819487 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:47:19.819522 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:47:19.825155 (+ 5633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:47:19.825852 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:47:19.825860 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:47:19.825869 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:47:19.826086 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:47:19.826484 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:47:19.826489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:47:19.826493 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:47:19.827164 (+ 671us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:47:19.829129 (+ 1965us) server_negotiation.cc:1035] Waiting for connection context
0810 00:47:19.829370 (+ 241us) server_negotiation.cc:299] Negotiation successful
0810 00:47:19.829670 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1559,"thread_start_us":1306,"threads_started":1}
I20260810 00:47:19.841718 13027 raft_consensus.cc:348] T 00000000000000000000000000000000 P dff4912c85064f6784b00d6b6fb06f3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff4912c85064f6784b00d6b6fb06f3f" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 45435 } }
I20260810 00:47:19.842306 13027 raft_consensus.cc:374] T 00000000000000000000000000000000 P dff4912c85064f6784b00d6b6fb06f3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:47:19.842437 13027 raft_consensus.cc:729] T 00000000000000000000000000000000 P dff4912c85064f6784b00d6b6fb06f3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff4912c85064f6784b00d6b6fb06f3f, State: Initialized, Role: FOLLOWER
I20260810 00:47:19.843458 13027 consensus_queue.cc:260] T 00000000000000000000000000000000 P dff4912c85064f6784b00d6b6fb06f3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff4912c85064f6784b00d6b6fb06f3f" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 45435 } }
I20260810 00:47:19.843945 13027 raft_consensus.cc:388] T 00000000000000000000000000000000 P dff4912c85064f6784b00d6b6fb06f3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:47:19.844198 13027 raft_consensus.cc:482] T 00000000000000000000000000000000 P dff4912c85064f6784b00d6b6fb06f3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:47:19.844511 13027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dff4912c85064f6784b00d6b6fb06f3f [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:47:19.848587 13027 raft_consensus.cc:504] T 00000000000000000000000000000000 P dff4912c85064f6784b00d6b6fb06f3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff4912c85064f6784b00d6b6fb06f3f" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 45435 } }
I20260810 00:47:19.849354 13027 leader_election.cc:302] T 00000000000000000000000000000000 P dff4912c85064f6784b00d6b6fb06f3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dff4912c85064f6784b00d6b6fb06f3f; no voters:
I20260810 00:47:19.850519 13027 leader_election.cc:288] T 00000000000000000000000000000000 P dff4912c85064f6784b00d6b6fb06f3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:47:19.850668 13033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dff4912c85064f6784b00d6b6fb06f3f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:47:19.853969 13027 sys_catalog.cc:564] T 00000000000000000000000000000000 P dff4912c85064f6784b00d6b6fb06f3f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:47:19.858100 13033 raft_consensus.cc:686] T 00000000000000000000000000000000 P dff4912c85064f6784b00d6b6fb06f3f [term 1 LEADER]: Becoming Leader. State: Replica: dff4912c85064f6784b00d6b6fb06f3f, State: Running, Role: LEADER
I20260810 00:47:19.859150 13033 consensus_queue.cc:237] T 00000000000000000000000000000000 P dff4912c85064f6784b00d6b6fb06f3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff4912c85064f6784b00d6b6fb06f3f" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 45435 } }
I20260810 00:47:19.863440 13035 sys_catalog.cc:455] T 00000000000000000000000000000000 P dff4912c85064f6784b00d6b6fb06f3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader dff4912c85064f6784b00d6b6fb06f3f. Latest consensus state: current_term: 1 leader_uuid: "dff4912c85064f6784b00d6b6fb06f3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff4912c85064f6784b00d6b6fb06f3f" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 45435 } } }
I20260810 00:47:19.864360 13035 sys_catalog.cc:458] T 00000000000000000000000000000000 P dff4912c85064f6784b00d6b6fb06f3f [sys.catalog]: This master's current role is: LEADER
I20260810 00:47:19.864616 13034 sys_catalog.cc:455] T 00000000000000000000000000000000 P dff4912c85064f6784b00d6b6fb06f3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dff4912c85064f6784b00d6b6fb06f3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff4912c85064f6784b00d6b6fb06f3f" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 45435 } } }
I20260810 00:47:19.865326 13034 sys_catalog.cc:458] T 00000000000000000000000000000000 P dff4912c85064f6784b00d6b6fb06f3f [sys.catalog]: This master's current role is: LEADER
I20260810 00:47:19.883926 13041 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:47:19.894372 13041 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:47:19.910989 13041 catalog_manager.cc:1347] Generated new cluster ID: bb2c7b3918ec4bf28b6a4f03e50bda54
I20260810 00:47:19.911221 13041 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:47:19.946624 13041 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:47:19.949553 13041 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:47:19.960270 13027 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 00:47:19.998852 13041 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dff4912c85064f6784b00d6b6fb06f3f: Generated new TSK 0
I20260810 00:47:20.000506 13041 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.6.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab.
Entry for principal kudu/127.28.6.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab.
I20260810 00:47:20.881235 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.6.20:59437/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.1:0
--local_ip_for_outbound_sockets=127.28.6.1
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:45435
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:42411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 00:47:21 dist-test-slave-1nl6 krb5kdc[12269](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786322841, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 00:47:21.623528 13087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:47:21.624019 13087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:47:21.624292 13087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:47:21.639555 13087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:47:21.639793 13087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:47:21.640163 13087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.1
I20260810 00:47:21.660411 13087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:42411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.6.20:59437/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:45435
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.1
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:47:21.662405 13087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:47:21.664173 13087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:47:21.695487 13100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:47:21.706071 13101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:47:21.710224 13087 server_base.cc:1034] running on GCE node
W20260810 00:47:21.711107 13105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:47:21.712061 13087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:47:21.723338 13087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:47:21.724753 13087 hybrid_clock.cc:648] HybridClock initialized: now 1786322841724508 us; error 152 us; skew 500 ppm
I20260810 00:47:21.725392 13087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:47:21 dist-test-slave-1nl6 krb5kdc[12269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322841, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 10 00:47:21 dist-test-slave-1nl6 krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322841, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for HTTP/127.28.6.20@KRBTEST.COM
I20260810 00:47:21.732404 13087 init.cc:377] Logged in from keytab as kudu/127.28.6.1@KRBTEST.COM (short username kudu)
I20260810 00:47:21.756513 13087 webserver.cc:460] Webserver started at http://127.28.6.1:41889/ using document root <none> and password file <none>
I20260810 00:47:21.757539 13087 fs_manager.cc:359] Metadata directory not provided
I20260810 00:47:21.757783 13087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:47:21.758258 13087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 00:47:21 dist-test-slave-1nl6 krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322841, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for HTTP/127.28.6.20@KRBTEST.COM
I20260810 00:47:21.793474 13087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-0/data/instance:
uuid: "6fb63dc8f45e47bea42de6b5b5051d96"
format_stamp: "Formatted at 2026-08-10 00:47:21 on dist-test-slave-1nl6"
server_key: "4c24289bfdc92794992d4ccaa5cb9720"
server_key_iv: "e983450da20137902bd37182333a652f"
server_key_version: "kuduclusterkey@0"
I20260810 00:47:21.794636 13087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance:
uuid: "6fb63dc8f45e47bea42de6b5b5051d96"
format_stamp: "Formatted at 2026-08-10 00:47:21 on dist-test-slave-1nl6"
server_key: "4c24289bfdc92794992d4ccaa5cb9720"
server_key_iv: "e983450da20137902bd37182333a652f"
server_key_version: "kuduclusterkey@0"
I20260810 00:47:21.803974 13087 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.003s sys 0.004s
I20260810 00:47:21.854871 13115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:47:21.857250 13087 fs_manager.cc:725] Time spent opening block manager: real 0.033s user 0.018s sys 0.012s
I20260810 00:47:21.857497 13087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-0/wal
uuid: "6fb63dc8f45e47bea42de6b5b5051d96"
format_stamp: "Formatted at 2026-08-10 00:47:21 on dist-test-slave-1nl6"
server_key: "4c24289bfdc92794992d4ccaa5cb9720"
server_key_iv: "e983450da20137902bd37182333a652f"
server_key_version: "kuduclusterkey@0"
I20260810 00:47:21.857856 13087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:47:21.907761 13087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:47:21.914718 13087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:47:21.915403 13087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:47:21.920717 13087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:47:21.923899 13087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:47:21.924084 13087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260810 00:47:21.924400 13087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:47:21.924649 13087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:47:22.153605 13087 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.1:44625
I20260810 00:47:22.153647 13277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.1:44625 every 8 connection(s)
I20260810 00:47:22.155882 13087 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
Aug 10 00:47:22 dist-test-slave-1nl6 krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322841, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for HTTP/127.28.6.20@KRBTEST.COM
I20260810 00:47:22.163179 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 13087
I20260810 00:47:22.163434 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance
I20260810 00:47:22.164323 28696 external_mini_cluster.cc:1428] Setting key 660e02b1d7e30dbeb30766e08fe1bd0a
WARNING: no policy specified for kudu/127.28.6.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.2@KRBTEST.COM" created.
Aug 10 00:47:22 dist-test-slave-1nl6 krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322841, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab.
Entry for principal kudu/127.28.6.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab.
W20260810 00:47:22.290678 28696 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.28.6.2: real 0.122s user 0.008s sys 0.015s
I20260810 00:47:22.291770 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.6.20:59437/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.2:0
--local_ip_for_outbound_sockets=127.28.6.2
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:45435
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:42411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 00:47:22.292833 13282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:47:22.214455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:41079 (local address 127.28.6.62:45435)
0810 00:47:22.229322 (+ 14867us) server_negotiation.cc:206] Beginning negotiation
0810 00:47:22.229328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:47:22.236729 (+ 7401us) server_negotiation.cc:407] Connection header received
0810 00:47:22.253455 (+ 16726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:47:22.253462 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:47:22.253585 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:47:22.253688 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:47:22.257937 (+ 4249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:47:22.259443 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:47:22.268379 (+ 8936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:47:22.269333 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:47:22.269437 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:47:22.281635 (+ 12198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:47:22.281642 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:47:22.281647 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:47:22.281782 (+ 135us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:47:22.287561 (+ 5779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:47:22.288974 (+ 1413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:47:22.288980 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:47:22.288984 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:47:22.289239 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:47:22.290593 (+ 1354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:47:22.290598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:47:22.290602 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:47:22.291015 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:47:22.291149 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 00:47:22.292144 (+ 995us) server_negotiation.cc:299] Negotiation successful
0810 00:47:22.292439 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14720,"thread_start_us":14528,"threads_started":1}
I20260810 00:47:22.295264 13281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:47:22.213413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:45435 (local address 127.28.6.1:41079)
0810 00:47:22.236527 (+ 23114us) negotiation.cc:107] Waiting for socket to connect
0810 00:47:22.236603 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0810 00:47:22.240686 (+ 4083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:47:22.254412 (+ 13726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:47:22.254425 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:47:22.255868 (+ 1443us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:47:22.257697 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:47:22.257728 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:47:22.260167 (+ 2439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:47:22.260172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:47:22.268187 (+ 8015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:47:22.268207 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:47:22.270063 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:47:22.270074 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:47:22.270501 (+ 427us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:47:22.275223 (+ 4722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:47:22.275258 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:47:22.279553 (+ 4295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:47:22.287829 (+ 8276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:47:22.287839 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:47:22.287859 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:47:22.288693 (+ 834us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:47:22.289449 (+ 756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:47:22.289455 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:47:22.289458 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:47:22.290450 (+ 992us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:47:22.291244 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:47:22.291252 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:47:22.291683 (+ 431us) client_negotiation.cc:769] Sending connection context
0810 00:47:22.293134 (+ 1451us) client_negotiation.cc:240] Negotiation successful
0810 00:47:22.293756 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17332,"thread_start_us":16134,"threads_started":1}
I20260810 00:47:22.297480 13278 heartbeater.cc:347] Connected to a master server at 127.28.6.62:45435
I20260810 00:47:22.297904 13278 heartbeater.cc:464] Registering TS with master...
I20260810 00:47:22.298945 13278 heartbeater.cc:511] Master 127.28.6.62:45435 requested a full tablet report, sending...
I20260810 00:47:22.301954 12974 ts_manager.cc:194] Registered new tserver with Master: 6fb63dc8f45e47bea42de6b5b5051d96 (127.28.6.1:44625)
I20260810 00:47:22.304561 12974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.1@KRBTEST.COM'} at 127.28.6.1:41079
W20260810 00:47:22.999759 13286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:47:23.000031 13286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:47:23.000154 13286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:47:23.007884 13286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:47:23.008011 13286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:47:23.008148 13286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.2
I20260810 00:47:23.018735 13286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:42411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.6.20:59437/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:45435
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.2
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:47:23.019994 13286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:47:23.021302 13286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:47:23.035240 13305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:47:23.035204 13307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:47:23.035331 13303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:47:23.036654 13286 server_base.cc:1034] running on GCE node
I20260810 00:47:23.037284 13286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:47:23.038210 13286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:47:23.039470 13286 hybrid_clock.cc:648] HybridClock initialized: now 1786322843039386 us; error 41 us; skew 500 ppm
I20260810 00:47:23.039829 13286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:47:23 dist-test-slave-1nl6 krb5kdc[12269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322843, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:47:23.043540 13286 init.cc:377] Logged in from keytab as kudu/127.28.6.2@KRBTEST.COM (short username kudu)
I20260810 00:47:23.045141 13286 webserver.cc:460] Webserver started at http://127.28.6.2:44491/ using document root <none> and password file <none>
I20260810 00:47:23.045941 13286 fs_manager.cc:359] Metadata directory not provided
I20260810 00:47:23.046123 13286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:47:23.046512 13286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 00:47:23 dist-test-slave-1nl6 krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322843, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for HTTP/127.28.6.20@KRBTEST.COM
I20260810 00:47:23.062413 13286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-1/data/instance:
uuid: "290f2289606b464ab216c0f64efd6b82"
format_stamp: "Formatted at 2026-08-10 00:47:23 on dist-test-slave-1nl6"
server_key: "87aa42e17cd3ee23e2ebadf9281f5ad1"
server_key_iv: "353b51027b6878f63a5b5c864c91040c"
server_key_version: "kuduclusterkey@0"
I20260810 00:47:23.063536 13286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance:
uuid: "290f2289606b464ab216c0f64efd6b82"
format_stamp: "Formatted at 2026-08-10 00:47:23 on dist-test-slave-1nl6"
server_key: "87aa42e17cd3ee23e2ebadf9281f5ad1"
server_key_iv: "353b51027b6878f63a5b5c864c91040c"
server_key_version: "kuduclusterkey@0"
I20260810 00:47:23.079293 13286 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.006s sys 0.009s
I20260810 00:47:23.094300 13316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:47:23.096211 13286 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260810 00:47:23.096426 13286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-1/wal
uuid: "290f2289606b464ab216c0f64efd6b82"
format_stamp: "Formatted at 2026-08-10 00:47:23 on dist-test-slave-1nl6"
server_key: "87aa42e17cd3ee23e2ebadf9281f5ad1"
server_key_iv: "353b51027b6878f63a5b5c864c91040c"
server_key_version: "kuduclusterkey@0"
I20260810 00:47:23.096665 13286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:47:23.111857 13286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:47:23.116127 13286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:47:23.116482 13286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:47:23.117868 13286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:47:23.119586 13286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:47:23.119704 13286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:47:23.119930 13286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:47:23.120069 13286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:47:23.188707 13286 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.2:33285
I20260810 00:47:23.188727 13489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.2:33285 every 8 connection(s)
I20260810 00:47:23.190178 13286 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
I20260810 00:47:23.197926 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 13286
I20260810 00:47:23.198100 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance
I20260810 00:47:23.198810 28696 external_mini_cluster.cc:1428] Setting key ad8068cb56f9c409c8c187d3023570fb
Aug 10 00:47:23 dist-test-slave-1nl6 krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322843, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:47:23.214406 13492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:47:23.193589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:34627 (local address 127.28.6.62:45435)
0810 00:47:23.194251 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0810 00:47:23.194258 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:47:23.195421 (+ 1163us) server_negotiation.cc:407] Connection header received
0810 00:47:23.197631 (+ 2210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:47:23.197637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:47:23.197726 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:47:23.197844 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:47:23.199534 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:47:23.200727 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:47:23.202992 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:47:23.203447 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:47:23.203581 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:47:23.207296 (+ 3715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:47:23.207301 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:47:23.207304 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:47:23.207341 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:47:23.210489 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:47:23.211369 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:47:23.211373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:47:23.211376 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:47:23.211611 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:47:23.212200 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:47:23.212205 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:47:23.212214 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:47:23.212546 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:47:23.212719 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 00:47:23.213676 (+ 957us) server_negotiation.cc:299] Negotiation successful
0810 00:47:23.213914 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":376,"threads_started":1}
I20260810 00:47:23.216184 13494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:47:23.194151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:45435 (local address 127.28.6.2:34627)
0810 00:47:23.195237 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0810 00:47:23.195291 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 00:47:23.197340 (+ 2049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:47:23.198090 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:47:23.198097 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:47:23.198574 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:47:23.199336 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:47:23.199356 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:47:23.200952 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:47:23.200958 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:47:23.202490 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:47:23.202502 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:47:23.203719 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:47:23.203728 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:47:23.203978 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:47:23.204713 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:47:23.204741 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:47:23.207062 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:47:23.210635 (+ 3573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:47:23.210643 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:47:23.210659 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:47:23.211229 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:47:23.211725 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:47:23.211731 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:47:23.211734 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:47:23.212067 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:47:23.212745 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:47:23.212755 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:47:23.213181 (+ 426us) client_negotiation.cc:769] Sending connection context
0810 00:47:23.214693 (+ 1512us) client_negotiation.cc:240] Negotiation successful
0810 00:47:23.215184 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"thread_start_us":509,"threads_started":1}
I20260810 00:47:23.217801 13490 heartbeater.cc:347] Connected to a master server at 127.28.6.62:45435
I20260810 00:47:23.218160 13490 heartbeater.cc:464] Registering TS with master...
I20260810 00:47:23.219043 13490 heartbeater.cc:511] Master 127.28.6.62:45435 requested a full tablet report, sending...
I20260810 00:47:23.220599 12974 ts_manager.cc:194] Registered new tserver with Master: 290f2289606b464ab216c0f64efd6b82 (127.28.6.2:33285)
I20260810 00:47:23.221455 12974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.2@KRBTEST.COM'} at 127.28.6.2:34627
WARNING: no policy specified for kudu/127.28.6.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab.
Entry for principal kudu/127.28.6.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab.
I20260810 00:47:23.268441 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.6.20:59437/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.3:0
--local_ip_for_outbound_sockets=127.28.6.3
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:45435
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:42411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 00:47:23.308884 13278 heartbeater.cc:503] Master 127.28.6.62:45435 was elected leader, sending a full tablet report...
W20260810 00:47:23.907106 13498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:47:23.907483 13498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:47:23.907680 13498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:47:23.916293 13498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:47:23.916465 13498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:47:23.916682 13498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.3
I20260810 00:47:23.928567 13498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:42411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.6.20:59437/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:45435
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.3
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:47:23.929932 13498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:47:23.931543 13498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:47:23.948683 13504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:47:23.948926 13505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:47:23.948906 13507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:47:23.951156 13498 server_base.cc:1034] running on GCE node
I20260810 00:47:23.951994 13498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:47:23.953055 13498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:47:23.954309 13498 hybrid_clock.cc:648] HybridClock initialized: now 1786322843954226 us; error 43 us; skew 500 ppm
I20260810 00:47:23.954735 13498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:47:23 dist-test-slave-1nl6 krb5kdc[12269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322843, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:47:23.963349 13498 init.cc:377] Logged in from keytab as kudu/127.28.6.3@KRBTEST.COM (short username kudu)
I20260810 00:47:23.965167 13498 webserver.cc:460] Webserver started at http://127.28.6.3:44831/ using document root <none> and password file <none>
I20260810 00:47:23.965920 13498 fs_manager.cc:359] Metadata directory not provided
I20260810 00:47:23.966099 13498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:47:23.966446 13498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 00:47:23 dist-test-slave-1nl6 krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322843, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for HTTP/127.28.6.20@KRBTEST.COM
I20260810 00:47:23.982504 13498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-2/data/instance:
uuid: "9bc2fccfad2d49df9c7cf054c382848f"
format_stamp: "Formatted at 2026-08-10 00:47:23 on dist-test-slave-1nl6"
server_key: "840364d3106dadc191dd96314d9811b9"
server_key_iv: "6d109b76a2a8b4077ef9bace00287d0f"
server_key_version: "kuduclusterkey@0"
I20260810 00:47:23.983505 13498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance:
uuid: "9bc2fccfad2d49df9c7cf054c382848f"
format_stamp: "Formatted at 2026-08-10 00:47:23 on dist-test-slave-1nl6"
server_key: "840364d3106dadc191dd96314d9811b9"
server_key_iv: "6d109b76a2a8b4077ef9bace00287d0f"
server_key_version: "kuduclusterkey@0"
I20260810 00:47:23.989611 13498 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 00:47:24.007858 13520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:47:24.009701 13498 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 00:47:24.009932 13498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-2/wal
uuid: "9bc2fccfad2d49df9c7cf054c382848f"
format_stamp: "Formatted at 2026-08-10 00:47:23 on dist-test-slave-1nl6"
server_key: "840364d3106dadc191dd96314d9811b9"
server_key_iv: "6d109b76a2a8b4077ef9bace00287d0f"
server_key_version: "kuduclusterkey@0"
I20260810 00:47:24.010281 13498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:47:24.029560 13498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:47:24.034560 13498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:47:24.034996 13498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:47:24.036471 13498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:47:24.038206 13498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:47:24.038333 13498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:47:24.038501 13498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:47:24.038596 13498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:47:24.102864 13498 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.3:42621
I20260810 00:47:24.103029 13678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.3:42621 every 8 connection(s)
I20260810 00:47:24.104442 13498 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
I20260810 00:47:24.112015 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 13498
I20260810 00:47:24.112229 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance
I20260810 00:47:24.112814 28696 external_mini_cluster.cc:1428] Setting key ae294ef93a4787ebbbf7bc1b67b23b93
Aug 10 00:47:24 dist-test-slave-1nl6 krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322843, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:47:24.129298 13682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:47:24.107541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:51897 (local address 127.28.6.62:45435)
0810 00:47:24.108305 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0810 00:47:24.108311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:47:24.109382 (+ 1071us) server_negotiation.cc:407] Connection header received
0810 00:47:24.111843 (+ 2461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:47:24.111849 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:47:24.111992 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:47:24.112172 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:47:24.114450 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:47:24.115897 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:47:24.117956 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:47:24.118543 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:47:24.118647 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:47:24.122628 (+ 3981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:47:24.122634 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:47:24.122637 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:47:24.122683 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:47:24.125656 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:47:24.126453 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:47:24.126457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:47:24.126459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:47:24.126588 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:47:24.127159 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:47:24.127163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:47:24.127165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:47:24.127594 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:47:24.127741 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 00:47:24.128705 (+ 964us) server_negotiation.cc:299] Negotiation successful
0810 00:47:24.128941 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":480,"threads_started":1}
I20260810 00:47:24.130370 13683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:47:24.107836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:45435 (local address 127.28.6.3:51897)
0810 00:47:24.109205 (+ 1369us) negotiation.cc:107] Waiting for socket to connect
0810 00:47:24.109258 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 00:47:24.111548 (+ 2290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:47:24.112519 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:47:24.112527 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:47:24.113098 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:47:24.114021 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:47:24.114066 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:47:24.116096 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:47:24.116101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:47:24.117783 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:47:24.117797 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:47:24.118805 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:47:24.118810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:47:24.119163 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:47:24.119877 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:47:24.119900 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:47:24.122389 (+ 2489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:47:24.125787 (+ 3398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:47:24.125793 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:47:24.125806 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:47:24.126343 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:47:24.126686 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:47:24.126689 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:47:24.126691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:47:24.127023 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:47:24.127782 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:47:24.127790 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:47:24.128223 (+ 433us) client_negotiation.cc:769] Sending connection context
0810 00:47:24.128583 (+ 360us) client_negotiation.cc:240] Negotiation successful
0810 00:47:24.129143 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1083,"thread_start_us":784,"threads_started":1}
I20260810 00:47:24.131839 13679 heartbeater.cc:347] Connected to a master server at 127.28.6.62:45435
I20260810 00:47:24.132133 13679 heartbeater.cc:464] Registering TS with master...
I20260810 00:47:24.132941 13679 heartbeater.cc:511] Master 127.28.6.62:45435 requested a full tablet report, sending...
I20260810 00:47:24.134441 12974 ts_manager.cc:194] Registered new tserver with Master: 9bc2fccfad2d49df9c7cf054c382848f (127.28.6.3:42621)
I20260810 00:47:24.135334 12974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.3@KRBTEST.COM'} at 127.28.6.3:51897
I20260810 00:47:24.142109 28696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 00:47:24.165261 13682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:47:24.148865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52866 (local address 127.28.6.62:45435)
0810 00:47:24.149060 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 00:47:24.149069 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:47:24.149970 (+ 901us) server_negotiation.cc:407] Connection header received
0810 00:47:24.150142 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:47:24.150145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:47:24.150217 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:47:24.150299 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:47:24.152523 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:47:24.153434 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:47:24.155068 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:47:24.155789 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:47:24.155966 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:47:24.159019 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:47:24.159025 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:47:24.159028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:47:24.159058 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:47:24.162158 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:47:24.162883 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:47:24.162889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:47:24.162893 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:47:24.163093 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:47:24.163574 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:47:24.163579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:47:24.163582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:47:24.163873 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:47:24.164173 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0810 00:47:24.164646 (+ 473us) server_negotiation.cc:299] Negotiation successful
0810 00:47:24.164907 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 00:47:24.169226 12974 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52866:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 00:47:24.178826 12974 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 00:47:24.221907 13699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:47:24.194674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:33285 (local address 127.0.0.1:42400)
0810 00:47:24.206765 (+ 12091us) negotiation.cc:107] Waiting for socket to connect
0810 00:47:24.206802 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 00:47:24.207045 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:47:24.211736 (+ 4691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:47:24.211743 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:47:24.211767 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:47:24.212726 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:47:24.212744 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:47:24.214652 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:47:24.214658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:47:24.218226 (+ 3568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:47:24.218239 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:47:24.220254 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:47:24.220261 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:47:24.220537 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:47:24.220587 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 00:47:24.220739 (+ 152us) client_negotiation.cc:240] Negotiation successful
0810 00:47:24.220970 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11740,"mutex_wait_us":34,"thread_start_us":12833,"threads_started":1}
I20260810 00:47:24.222005 13705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:47:24.199244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52978 (local address 127.28.6.3:42621)
0810 00:47:24.203310 (+ 4066us) server_negotiation.cc:206] Beginning negotiation
0810 00:47:24.203323 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 00:47:24.208782 (+ 5459us) server_negotiation.cc:407] Connection header received
0810 00:47:24.209147 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:47:24.209159 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:47:24.209491 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:47:24.209675 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:47:24.210844 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:47:24.212998 (+ 2154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:47:24.217697 (+ 4699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:47:24.219936 (+ 2239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:47:24.220064 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:47:24.220438 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0810 00:47:24.221366 (+ 928us) server_negotiation.cc:299] Negotiation successful
0810 00:47:24.221549 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3808,"thread_start_us":3530,"threads_started":1}
I20260810 00:47:24.222718 13700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:47:24.195189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42400 (local address 127.28.6.2:33285)
0810 00:47:24.210785 (+ 15596us) server_negotiation.cc:206] Beginning negotiation
0810 00:47:24.210794 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:47:24.210822 (+ 28us) server_negotiation.cc:407] Connection header received
0810 00:47:24.210955 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:47:24.210962 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:47:24.211191 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:47:24.211380 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:47:24.213097 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:47:24.214460 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:47:24.218421 (+ 3961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:47:24.219930 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:47:24.220088 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:47:24.220498 (+ 410us) server_negotiation.cc:1035] Waiting for connection context
0810 00:47:24.220784 (+ 286us) server_negotiation.cc:299] Negotiation successful
0810 00:47:24.220960 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15400,"thread_start_us":15200,"threads_started":1}
I20260810 00:47:24.222957 13701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:47:24.195492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50380 (local address 127.28.6.1:44625)
0810 00:47:24.210189 (+ 14697us) server_negotiation.cc:206] Beginning negotiation
0810 00:47:24.210201 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 00:47:24.210239 (+ 38us) server_negotiation.cc:407] Connection header received
0810 00:47:24.210401 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:47:24.210412 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:47:24.210634 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:47:24.211045 (+ 411us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:47:24.212220 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:47:24.214542 (+ 2322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:47:24.219191 (+ 4649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:47:24.220972 (+ 1781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:47:24.221079 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:47:24.221273 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0810 00:47:24.222329 (+ 1056us) server_negotiation.cc:299] Negotiation successful
0810 00:47:24.222494 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14332,"thread_start_us":13948,"threads_started":1}
I20260810 00:47:24.222761 13704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:47:24.198120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:42621 (local address 127.0.0.1:52978)
0810 00:47:24.208588 (+ 10468us) negotiation.cc:107] Waiting for socket to connect
0810 00:47:24.208610 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 00:47:24.208923 (+ 313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:47:24.209821 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:47:24.209829 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:47:24.209863 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:47:24.210551 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:47:24.210568 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:47:24.213216 (+ 2648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:47:24.213221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:47:24.217505 (+ 4284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:47:24.217520 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:47:24.220729 (+ 3209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:47:24.220737 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:47:24.220925 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:47:24.220956 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 00:47:24.221052 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 00:47:24.221189 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10295,"mutex_wait_us":137,"thread_start_us":10024,"threads_started":1}
I20260810 00:47:24.223649 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:47:24.195010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:44625 (local address 127.0.0.1:50380)
0810 00:47:24.207931 (+ 12921us) negotiation.cc:107] Waiting for socket to connect
0810 00:47:24.207958 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 00:47:24.208115 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:47:24.211018 (+ 2903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:47:24.211022 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:47:24.211048 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:47:24.211760 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:47:24.211772 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:47:24.214754 (+ 2982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:47:24.214763 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:47:24.218926 (+ 4163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:47:24.218942 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:47:24.221476 (+ 2534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:47:24.221482 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:47:24.221701 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:47:24.221745 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 00:47:24.221845 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 00:47:24.221969 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12730,"mutex_wait_us":131,"thread_start_us":11173,"threads_started":1}
I20260810 00:47:24.224061 13490 heartbeater.cc:503] Master 127.28.6.62:45435 was elected leader, sending a full tablet report...
I20260810 00:47:24.227149 13593 tablet_service.cc:1467] Processing CreateTablet for tablet 35eaf4a628d847f4bed182363e7a393c (DEFAULT_TABLE table=test-table [id=16b83f7bf23748de96e0e662ddd78253]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:47:24.227810 13187 tablet_service.cc:1467] Processing CreateTablet for tablet 35eaf4a628d847f4bed182363e7a393c (DEFAULT_TABLE table=test-table [id=16b83f7bf23748de96e0e662ddd78253]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:47:24.229202 13593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35eaf4a628d847f4bed182363e7a393c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:47:24.229660 13187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35eaf4a628d847f4bed182363e7a393c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:47:24.230016 13396 tablet_service.cc:1467] Processing CreateTablet for tablet 35eaf4a628d847f4bed182363e7a393c (DEFAULT_TABLE table=test-table [id=16b83f7bf23748de96e0e662ddd78253]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:47:24.231994 13396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35eaf4a628d847f4bed182363e7a393c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:47:24.247242 13706 tablet_bootstrap.cc:492] T 35eaf4a628d847f4bed182363e7a393c P 290f2289606b464ab216c0f64efd6b82: Bootstrap starting.
I20260810 00:47:24.252123 13707 tablet_bootstrap.cc:492] T 35eaf4a628d847f4bed182363e7a393c P 9bc2fccfad2d49df9c7cf054c382848f: Bootstrap starting.
I20260810 00:47:24.252457 13706 tablet_bootstrap.cc:654] T 35eaf4a628d847f4bed182363e7a393c P 290f2289606b464ab216c0f64efd6b82: Neither blocks nor log segments found. Creating new log.
I20260810 00:47:24.253906 13708 tablet_bootstrap.cc:492] T 35eaf4a628d847f4bed182363e7a393c P 6fb63dc8f45e47bea42de6b5b5051d96: Bootstrap starting.
I20260810 00:47:24.255260 13706 log.cc:824] T 35eaf4a628d847f4bed182363e7a393c P 290f2289606b464ab216c0f64efd6b82: Log is configured to *not* fsync() on all Append() calls
I20260810 00:47:24.257445 13707 tablet_bootstrap.cc:654] T 35eaf4a628d847f4bed182363e7a393c P 9bc2fccfad2d49df9c7cf054c382848f: Neither blocks nor log segments found. Creating new log.
I20260810 00:47:24.258996 13707 log.cc:824] T 35eaf4a628d847f4bed182363e7a393c P 9bc2fccfad2d49df9c7cf054c382848f: Log is configured to *not* fsync() on all Append() calls
I20260810 00:47:24.259187 13708 tablet_bootstrap.cc:654] T 35eaf4a628d847f4bed182363e7a393c P 6fb63dc8f45e47bea42de6b5b5051d96: Neither blocks nor log segments found. Creating new log.
I20260810 00:47:24.260143 13706 tablet_bootstrap.cc:492] T 35eaf4a628d847f4bed182363e7a393c P 290f2289606b464ab216c0f64efd6b82: No bootstrap required, opened a new log
I20260810 00:47:24.260497 13706 ts_tablet_manager.cc:1397] T 35eaf4a628d847f4bed182363e7a393c P 290f2289606b464ab216c0f64efd6b82: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260810 00:47:24.261055 13708 log.cc:824] T 35eaf4a628d847f4bed182363e7a393c P 6fb63dc8f45e47bea42de6b5b5051d96: Log is configured to *not* fsync() on all Append() calls
I20260810 00:47:24.263511 13707 tablet_bootstrap.cc:492] T 35eaf4a628d847f4bed182363e7a393c P 9bc2fccfad2d49df9c7cf054c382848f: No bootstrap required, opened a new log
I20260810 00:47:24.263957 13707 ts_tablet_manager.cc:1397] T 35eaf4a628d847f4bed182363e7a393c P 9bc2fccfad2d49df9c7cf054c382848f: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.002s
I20260810 00:47:24.264683 13708 tablet_bootstrap.cc:492] T 35eaf4a628d847f4bed182363e7a393c P 6fb63dc8f45e47bea42de6b5b5051d96: No bootstrap required, opened a new log
I20260810 00:47:24.265039 13708 ts_tablet_manager.cc:1397] T 35eaf4a628d847f4bed182363e7a393c P 6fb63dc8f45e47bea42de6b5b5051d96: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260810 00:47:24.276121 13707 raft_consensus.cc:348] T 35eaf4a628d847f4bed182363e7a393c P 9bc2fccfad2d49df9c7cf054c382848f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f2289606b464ab216c0f64efd6b82" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 33285 } } peers { permanent_uuid: "6fb63dc8f45e47bea42de6b5b5051d96" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44625 } } peers { permanent_uuid: "9bc2fccfad2d49df9c7cf054c382848f" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 42621 } }
I20260810 00:47:24.276223 13706 raft_consensus.cc:348] T 35eaf4a628d847f4bed182363e7a393c P 290f2289606b464ab216c0f64efd6b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f2289606b464ab216c0f64efd6b82" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 33285 } } peers { permanent_uuid: "6fb63dc8f45e47bea42de6b5b5051d96" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44625 } } peers { permanent_uuid: "9bc2fccfad2d49df9c7cf054c382848f" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 42621 } }
I20260810 00:47:24.276536 13707 raft_consensus.cc:374] T 35eaf4a628d847f4bed182363e7a393c P 9bc2fccfad2d49df9c7cf054c382848f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:47:24.276685 13707 raft_consensus.cc:729] T 35eaf4a628d847f4bed182363e7a393c P 9bc2fccfad2d49df9c7cf054c382848f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc2fccfad2d49df9c7cf054c382848f, State: Initialized, Role: FOLLOWER
I20260810 00:47:24.276703 13706 raft_consensus.cc:374] T 35eaf4a628d847f4bed182363e7a393c P 290f2289606b464ab216c0f64efd6b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:47:24.276847 13706 raft_consensus.cc:729] T 35eaf4a628d847f4bed182363e7a393c P 290f2289606b464ab216c0f64efd6b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 290f2289606b464ab216c0f64efd6b82, State: Initialized, Role: FOLLOWER
I20260810 00:47:24.277100 13708 raft_consensus.cc:348] T 35eaf4a628d847f4bed182363e7a393c P 6fb63dc8f45e47bea42de6b5b5051d96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f2289606b464ab216c0f64efd6b82" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 33285 } } peers { permanent_uuid: "6fb63dc8f45e47bea42de6b5b5051d96" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44625 } } peers { permanent_uuid: "9bc2fccfad2d49df9c7cf054c382848f" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 42621 } }
I20260810 00:47:24.277572 13708 raft_consensus.cc:374] T 35eaf4a628d847f4bed182363e7a393c P 6fb63dc8f45e47bea42de6b5b5051d96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:47:24.277403 13707 consensus_queue.cc:260] T 35eaf4a628d847f4bed182363e7a393c P 9bc2fccfad2d49df9c7cf054c382848f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f2289606b464ab216c0f64efd6b82" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 33285 } } peers { permanent_uuid: "6fb63dc8f45e47bea42de6b5b5051d96" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44625 } } peers { permanent_uuid: "9bc2fccfad2d49df9c7cf054c382848f" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 42621 } }
I20260810 00:47:24.277716 13708 raft_consensus.cc:729] T 35eaf4a628d847f4bed182363e7a393c P 6fb63dc8f45e47bea42de6b5b5051d96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fb63dc8f45e47bea42de6b5b5051d96, State: Initialized, Role: FOLLOWER
I20260810 00:47:24.277631 13706 consensus_queue.cc:260] T 35eaf4a628d847f4bed182363e7a393c P 290f2289606b464ab216c0f64efd6b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f2289606b464ab216c0f64efd6b82" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 33285 } } peers { permanent_uuid: "6fb63dc8f45e47bea42de6b5b5051d96" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44625 } } peers { permanent_uuid: "9bc2fccfad2d49df9c7cf054c382848f" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 42621 } }
I20260810 00:47:24.278518 13708 consensus_queue.cc:260] T 35eaf4a628d847f4bed182363e7a393c P 6fb63dc8f45e47bea42de6b5b5051d96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f2289606b464ab216c0f64efd6b82" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 33285 } } peers { permanent_uuid: "6fb63dc8f45e47bea42de6b5b5051d96" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44625 } } peers { permanent_uuid: "9bc2fccfad2d49df9c7cf054c382848f" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 42621 } }
I20260810 00:47:24.279006 13679 heartbeater.cc:503] Master 127.28.6.62:45435 was elected leader, sending a full tablet report...
I20260810 00:47:24.279910 13707 ts_tablet_manager.cc:1428] T 35eaf4a628d847f4bed182363e7a393c P 9bc2fccfad2d49df9c7cf054c382848f: Time spent starting tablet: real 0.016s user 0.010s sys 0.007s
I20260810 00:47:24.282467 13706 ts_tablet_manager.cc:1428] T 35eaf4a628d847f4bed182363e7a393c P 290f2289606b464ab216c0f64efd6b82: Time spent starting tablet: real 0.022s user 0.011s sys 0.009s
I20260810 00:47:24.285560 13708 ts_tablet_manager.cc:1428] T 35eaf4a628d847f4bed182363e7a393c P 6fb63dc8f45e47bea42de6b5b5051d96: Time spent starting tablet: real 0.020s user 0.003s sys 0.017s
W20260810 00:47:24.356648 13681 tablet.cc:2369] T 35eaf4a628d847f4bed182363e7a393c P 9bc2fccfad2d49df9c7cf054c382848f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:47:24.368983 13714 raft_consensus.cc:482] T 35eaf4a628d847f4bed182363e7a393c P 9bc2fccfad2d49df9c7cf054c382848f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:47:24.369406 13714 raft_consensus.cc:504] T 35eaf4a628d847f4bed182363e7a393c P 9bc2fccfad2d49df9c7cf054c382848f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f2289606b464ab216c0f64efd6b82" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 33285 } } peers { permanent_uuid: "6fb63dc8f45e47bea42de6b5b5051d96" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44625 } } peers { permanent_uuid: "9bc2fccfad2d49df9c7cf054c382848f" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 42621 } }
I20260810 00:47:24.371605 13714 leader_election.cc:288] T 35eaf4a628d847f4bed182363e7a393c P 9bc2fccfad2d49df9c7cf054c382848f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 290f2289606b464ab216c0f64efd6b82 (127.28.6.2:33285), 6fb63dc8f45e47bea42de6b5b5051d96 (127.28.6.1:44625)
I20260810 00:47:24.381292 13700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:47:24.372244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:40397 (local address 127.28.6.2:33285)
0810 00:47:24.372444 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 00:47:24.372451 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:47:24.373382 (+ 931us) server_negotiation.cc:407] Connection header received
0810 00:47:24.373578 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:47:24.373584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:47:24.373709 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:47:24.373849 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:47:24.374697 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:47:24.375715 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:47:24.378283 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:47:24.379943 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:47:24.380128 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:47:24.380264 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 00:47:24.380823 (+ 559us) server_negotiation.cc:299] Negotiation successful
0810 00:47:24.380957 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 00:47:24.381309 13701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:47:24.372122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:39441 (local address 127.28.6.1:44625)
0810 00:47:24.372369 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0810 00:47:24.372376 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:47:24.372565 (+ 189us) server_negotiation.cc:407] Connection header received
0810 00:47:24.372754 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:47:24.372763 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:47:24.372870 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:47:24.372990 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:47:24.373495 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:47:24.374690 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:47:24.378468 (+ 3778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:47:24.380142 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:47:24.380265 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:47:24.380367 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 00:47:24.380823 (+ 456us) server_negotiation.cc:299] Negotiation successful
0810 00:47:24.380957 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260810 00:47:24.381413 13683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:47:24.371979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:44625 (local address 127.28.6.3:39441)
0810 00:47:24.372442 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0810 00:47:24.372465 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 00:47:24.372596 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:47:24.373047 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:47:24.373052 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:47:24.373105 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:47:24.373356 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:47:24.373370 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:47:24.374843 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:47:24.374847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:47:24.378270 (+ 3423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:47:24.378290 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:47:24.380286 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:47:24.380292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:47:24.380598 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:47:24.380674 (+ 76us) client_negotiation.cc:769] Sending connection context
0810 00:47:24.380770 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 00:47:24.381002 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":127}
I20260810 00:47:24.381584 13720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:47:24.372376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:33285 (local address 127.28.6.3:40397)
0810 00:47:24.373273 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0810 00:47:24.373293 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 00:47:24.373423 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:47:24.373860 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:47:24.373863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:47:24.373882 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:47:24.374519 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:47:24.374529 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:47:24.375902 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:47:24.375906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:47:24.378071 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:47:24.378084 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:47:24.380188 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:47:24.380192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:47:24.380581 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:47:24.380658 (+ 77us) client_negotiation.cc:769] Sending connection context
0810 00:47:24.380765 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 00:47:24.381002 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"mutex_wait_us":140,"thread_start_us":516,"threads_started":1}
I20260810 00:47:24.382800 13217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35eaf4a628d847f4bed182363e7a393c" candidate_uuid: "9bc2fccfad2d49df9c7cf054c382848f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fb63dc8f45e47bea42de6b5b5051d96" is_pre_election: true
I20260810 00:47:24.383510 13217 raft_consensus.cc:2454] T 35eaf4a628d847f4bed182363e7a393c P 6fb63dc8f45e47bea42de6b5b5051d96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc2fccfad2d49df9c7cf054c382848f in term 0.
I20260810 00:47:24.383445 13426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35eaf4a628d847f4bed182363e7a393c" candidate_uuid: "9bc2fccfad2d49df9c7cf054c382848f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "290f2289606b464ab216c0f64efd6b82" is_pre_election: true
I20260810 00:47:24.384238 13426 raft_consensus.cc:2454] T 35eaf4a628d847f4bed182363e7a393c P 290f2289606b464ab216c0f64efd6b82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc2fccfad2d49df9c7cf054c382848f in term 0.
I20260810 00:47:24.384336 13522 leader_election.cc:302] T 35eaf4a628d847f4bed182363e7a393c P 9bc2fccfad2d49df9c7cf054c382848f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fb63dc8f45e47bea42de6b5b5051d96, 9bc2fccfad2d49df9c7cf054c382848f; no voters:
I20260810 00:47:24.384882 13714 raft_consensus.cc:2781] T 35eaf4a628d847f4bed182363e7a393c P 9bc2fccfad2d49df9c7cf054c382848f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 00:47:24.385113 13714 raft_consensus.cc:482] T 35eaf4a628d847f4bed182363e7a393c P 9bc2fccfad2d49df9c7cf054c382848f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 00:47:24.385289 13714 raft_consensus.cc:3037] T 35eaf4a628d847f4bed182363e7a393c P 9bc2fccfad2d49df9c7cf054c382848f [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:47:24.388178 13714 raft_consensus.cc:504] T 35eaf4a628d847f4bed182363e7a393c P 9bc2fccfad2d49df9c7cf054c382848f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f2289606b464ab216c0f64efd6b82" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 33285 } } peers { permanent_uuid: "6fb63dc8f45e47bea42de6b5b5051d96" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44625 } } peers { permanent_uuid: "9bc2fccfad2d49df9c7cf054c382848f" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 42621 } }
I20260810 00:47:24.389130 13714 leader_election.cc:288] T 35eaf4a628d847f4bed182363e7a393c P 9bc2fccfad2d49df9c7cf054c382848f [CANDIDATE]: Term 1 election: Requested vote from peers 290f2289606b464ab216c0f64efd6b82 (127.28.6.2:33285), 6fb63dc8f45e47bea42de6b5b5051d96 (127.28.6.1:44625)
I20260810 00:47:24.389505 13426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35eaf4a628d847f4bed182363e7a393c" candidate_uuid: "9bc2fccfad2d49df9c7cf054c382848f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "290f2289606b464ab216c0f64efd6b82"
I20260810 00:47:24.389780 13426 raft_consensus.cc:3037] T 35eaf4a628d847f4bed182363e7a393c P 290f2289606b464ab216c0f64efd6b82 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:47:24.389765 13217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35eaf4a628d847f4bed182363e7a393c" candidate_uuid: "9bc2fccfad2d49df9c7cf054c382848f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fb63dc8f45e47bea42de6b5b5051d96"
I20260810 00:47:24.389986 13217 raft_consensus.cc:3037] T 35eaf4a628d847f4bed182363e7a393c P 6fb63dc8f45e47bea42de6b5b5051d96 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:47:24.394495 13426 raft_consensus.cc:2454] T 35eaf4a628d847f4bed182363e7a393c P 290f2289606b464ab216c0f64efd6b82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc2fccfad2d49df9c7cf054c382848f in term 1.
I20260810 00:47:24.394551 13217 raft_consensus.cc:2454] T 35eaf4a628d847f4bed182363e7a393c P 6fb63dc8f45e47bea42de6b5b5051d96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc2fccfad2d49df9c7cf054c382848f in term 1.
I20260810 00:47:24.395092 13522 leader_election.cc:302] T 35eaf4a628d847f4bed182363e7a393c P 9bc2fccfad2d49df9c7cf054c382848f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fb63dc8f45e47bea42de6b5b5051d96, 9bc2fccfad2d49df9c7cf054c382848f; no voters:
I20260810 00:47:24.395412 13714 raft_consensus.cc:2781] T 35eaf4a628d847f4bed182363e7a393c P 9bc2fccfad2d49df9c7cf054c382848f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:47:24.396268 13714 raft_consensus.cc:686] T 35eaf4a628d847f4bed182363e7a393c P 9bc2fccfad2d49df9c7cf054c382848f [term 1 LEADER]: Becoming Leader. State: Replica: 9bc2fccfad2d49df9c7cf054c382848f, State: Running, Role: LEADER
I20260810 00:47:24.397163 13714 consensus_queue.cc:237] T 35eaf4a628d847f4bed182363e7a393c P 9bc2fccfad2d49df9c7cf054c382848f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f2289606b464ab216c0f64efd6b82" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 33285 } } peers { permanent_uuid: "6fb63dc8f45e47bea42de6b5b5051d96" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44625 } } peers { permanent_uuid: "9bc2fccfad2d49df9c7cf054c382848f" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 42621 } }
I20260810 00:47:24.402073 12971 catalog_manager.cc:5495] T 35eaf4a628d847f4bed182363e7a393c P 9bc2fccfad2d49df9c7cf054c382848f reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bc2fccfad2d49df9c7cf054c382848f (127.28.6.3). New cstate: current_term: 1 leader_uuid: "9bc2fccfad2d49df9c7cf054c382848f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290f2289606b464ab216c0f64efd6b82" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 33285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6fb63dc8f45e47bea42de6b5b5051d96" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bc2fccfad2d49df9c7cf054c382848f" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 42621 } health_report { overall_health: HEALTHY } } }
W20260810 00:47:24.443102 13491 tablet.cc:2369] T 35eaf4a628d847f4bed182363e7a393c P 290f2289606b464ab216c0f64efd6b82: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:47:24.457089 13705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:47:24.450919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52994 (local address 127.28.6.3:42621)
0810 00:47:24.451125 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 00:47:24.451132 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:47:24.451179 (+ 47us) server_negotiation.cc:407] Connection header received
0810 00:47:24.451431 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:47:24.451435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:47:24.451534 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:47:24.451690 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:47:24.452144 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:47:24.452934 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:47:24.454975 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:47:24.455446 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:47:24.455549 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:47:24.456045 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:47:24.456534 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:47:24.456639 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 00:47:24.456728 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 00:47:24.456791 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260810 00:47:24.460549 13280 tablet.cc:2369] T 35eaf4a628d847f4bed182363e7a393c P 6fb63dc8f45e47bea42de6b5b5051d96: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:47:24.466681 13217 raft_consensus.cc:1261] T 35eaf4a628d847f4bed182363e7a393c P 6fb63dc8f45e47bea42de6b5b5051d96 [term 1 FOLLOWER]: Refusing update from remote peer 9bc2fccfad2d49df9c7cf054c382848f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:47:24.466935 13426 raft_consensus.cc:1261] T 35eaf4a628d847f4bed182363e7a393c P 290f2289606b464ab216c0f64efd6b82 [term 1 FOLLOWER]: Refusing update from remote peer 9bc2fccfad2d49df9c7cf054c382848f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:47:24.467911 13724 consensus_queue.cc:1035] T 35eaf4a628d847f4bed182363e7a393c P 9bc2fccfad2d49df9c7cf054c382848f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fb63dc8f45e47bea42de6b5b5051d96" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 44625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:47:24.468390 13714 consensus_queue.cc:1035] T 35eaf4a628d847f4bed182363e7a393c P 9bc2fccfad2d49df9c7cf054c382848f [LEADER]: Connected to new peer: Peer: permanent_uuid: "290f2289606b464ab216c0f64efd6b82" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 33285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:47:24.500626 12971 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:47:24.500962 12971 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:47:24.504969 12971 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dff4912c85064f6784b00d6b6fb06f3f: Sending DeleteTablet for 3 replicas of tablet 35eaf4a628d847f4bed182363e7a393c
I20260810 00:47:24.506625 13187 tablet_service.cc:1514] Processing DeleteTablet for tablet 35eaf4a628d847f4bed182363e7a393c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:47:24 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:50380
I20260810 00:47:24.506722 13593 tablet_service.cc:1514] Processing DeleteTablet for tablet 35eaf4a628d847f4bed182363e7a393c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:47:24 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:52978
I20260810 00:47:24.508942 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 13087
I20260810 00:47:24.510257 13396 tablet_service.cc:1514] Processing DeleteTablet for tablet 35eaf4a628d847f4bed182363e7a393c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:47:24 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:42400
I20260810 00:47:24.517339 13747 tablet_replica.cc:330] T 35eaf4a628d847f4bed182363e7a393c P 290f2289606b464ab216c0f64efd6b82: stopping tablet replica
I20260810 00:47:24.517887 13747 raft_consensus.cc:2229] T 35eaf4a628d847f4bed182363e7a393c P 290f2289606b464ab216c0f64efd6b82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:47:24.518643 13747 raft_consensus.cc:2258] T 35eaf4a628d847f4bed182363e7a393c P 290f2289606b464ab216c0f64efd6b82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:47:24.520372 13744 tablet_replica.cc:330] T 35eaf4a628d847f4bed182363e7a393c P 6fb63dc8f45e47bea42de6b5b5051d96: stopping tablet replica
I20260810 00:47:24.521107 13744 raft_consensus.cc:2229] T 35eaf4a628d847f4bed182363e7a393c P 6fb63dc8f45e47bea42de6b5b5051d96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:47:24.521364 13747 ts_tablet_manager.cc:1905] T 35eaf4a628d847f4bed182363e7a393c P 290f2289606b464ab216c0f64efd6b82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:47:24.522819 13744 raft_consensus.cc:2258] T 35eaf4a628d847f4bed182363e7a393c P 6fb63dc8f45e47bea42de6b5b5051d96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:47:24.525907 13742 tablet_replica.cc:330] T 35eaf4a628d847f4bed182363e7a393c P 9bc2fccfad2d49df9c7cf054c382848f: stopping tablet replica
I20260810 00:47:24.526806 13742 raft_consensus.cc:2229] T 35eaf4a628d847f4bed182363e7a393c P 9bc2fccfad2d49df9c7cf054c382848f [term 1 LEADER]: Raft consensus shutting down.
I20260810 00:47:24.528350 13744 ts_tablet_manager.cc:1905] T 35eaf4a628d847f4bed182363e7a393c P 6fb63dc8f45e47bea42de6b5b5051d96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:47:24.530395 13747 ts_tablet_manager.cc:1918] T 35eaf4a628d847f4bed182363e7a393c P 290f2289606b464ab216c0f64efd6b82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:47:24.530740 13747 log.cc:1196] T 35eaf4a628d847f4bed182363e7a393c P 290f2289606b464ab216c0f64efd6b82: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-1/wal/wals/35eaf4a628d847f4bed182363e7a393c
I20260810 00:47:24.530787 13742 raft_consensus.cc:2258] T 35eaf4a628d847f4bed182363e7a393c P 9bc2fccfad2d49df9c7cf054c382848f [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 10 00:47:24 dist-test-slave-1nl6 krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.1@KRBTEST.COM
I20260810 00:47:24.531466 13747 ts_tablet_manager.cc:1939] T 35eaf4a628d847f4bed182363e7a393c P 290f2289606b464ab216c0f64efd6b82: Deleting consensus metadata
I20260810 00:47:24.533969 12948 catalog_manager.cc:4841] TS 290f2289606b464ab216c0f64efd6b82 (127.28.6.2:33285): tablet 35eaf4a628d847f4bed182363e7a393c (table test-table [id=16b83f7bf23748de96e0e662ddd78253]) successfully deleted
I20260810 00:47:24.536139 13742 ts_tablet_manager.cc:1905] T 35eaf4a628d847f4bed182363e7a393c P 9bc2fccfad2d49df9c7cf054c382848f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:47:24.539307 13744 ts_tablet_manager.cc:1918] T 35eaf4a628d847f4bed182363e7a393c P 6fb63dc8f45e47bea42de6b5b5051d96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:47:24.539638 13744 log.cc:1196] T 35eaf4a628d847f4bed182363e7a393c P 6fb63dc8f45e47bea42de6b5b5051d96: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-0/wal/wals/35eaf4a628d847f4bed182363e7a393c
I20260810 00:47:24.540944 13744 ts_tablet_manager.cc:1939] T 35eaf4a628d847f4bed182363e7a393c P 6fb63dc8f45e47bea42de6b5b5051d96: Deleting consensus metadata
I20260810 00:47:24.541630 13701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:47:24.510951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50384 (local address 127.28.6.1:44625)
0810 00:47:24.511185 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0810 00:47:24.511204 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 00:47:24.517625 (+ 6421us) server_negotiation.cc:407] Connection header received
0810 00:47:24.517913 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:47:24.517922 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:47:24.518075 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:47:24.518163 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:47:24.520583 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:47:24.522256 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:47:24.525643 (+ 3387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:47:24.526562 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:47:24.526687 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:47:24.532506 (+ 5819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:47:24.532526 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:47:24.532544 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:47:24.532586 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:47:24.537576 (+ 4990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:47:24.538527 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:47:24.538536 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:47:24.538540 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:47:24.538705 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:47:24.539268 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:47:24.539273 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:47:24.539278 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:47:24.539946 (+ 668us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:47:24.540304 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0810 00:47:24.540757 (+ 453us) server_negotiation.cc:299] Negotiation successful
0810 00:47:24.540979 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 00:47:24.543474 12946 catalog_manager.cc:4841] TS 6fb63dc8f45e47bea42de6b5b5051d96 (127.28.6.1:44625): tablet 35eaf4a628d847f4bed182363e7a393c (table test-table [id=16b83f7bf23748de96e0e662ddd78253]) successfully deleted
I20260810 00:47:24.543632 13272 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:47:24.546618 13742 ts_tablet_manager.cc:1918] T 35eaf4a628d847f4bed182363e7a393c P 9bc2fccfad2d49df9c7cf054c382848f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:47:24.546885 13742 log.cc:1196] T 35eaf4a628d847f4bed182363e7a393c P 9bc2fccfad2d49df9c7cf054c382848f: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786322687226232-28696-0/minicluster-data/ts-2/wal/wals/35eaf4a628d847f4bed182363e7a393c
I20260810 00:47:24.547571 13742 ts_tablet_manager.cc:1939] T 35eaf4a628d847f4bed182363e7a393c P 9bc2fccfad2d49df9c7cf054c382848f: Deleting consensus metadata
I20260810 00:47:24.549392 12945 catalog_manager.cc:4841] TS 9bc2fccfad2d49df9c7cf054c382848f (127.28.6.3:42621): tablet 35eaf4a628d847f4bed182363e7a393c (table test-table [id=16b83f7bf23748de96e0e662ddd78253]) successfully deleted
I20260810 00:47:24.638965 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 13087
I20260810 00:47:24.670992 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 13286
Aug 10 00:47:24 dist-test-slave-1nl6 krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.2@KRBTEST.COM
I20260810 00:47:24.691411 13700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:47:24.671893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42416 (local address 127.28.6.2:33285)
0810 00:47:24.672100 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 00:47:24.672107 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:47:24.672130 (+ 23us) server_negotiation.cc:407] Connection header received
0810 00:47:24.672315 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:47:24.672322 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:47:24.672412 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:47:24.672524 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:47:24.674216 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:47:24.675760 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:47:24.677177 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:47:24.677997 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:47:24.678125 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:47:24.682874 (+ 4749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:47:24.682893 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:47:24.682948 (+ 55us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:47:24.683008 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:47:24.687909 (+ 4901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:47:24.688559 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:47:24.688569 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:47:24.688573 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:47:24.688813 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:47:24.689220 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:47:24.689225 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:47:24.689229 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:47:24.689783 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:47:24.690138 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0810 00:47:24.690534 (+ 396us) server_negotiation.cc:299] Negotiation successful
0810 00:47:24.690766 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 00:47:24.692421 13483 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:47:24.781970 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 13286
I20260810 00:47:24.805986 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 13498
Aug 10 00:47:24 dist-test-slave-1nl6 krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.3@KRBTEST.COM
I20260810 00:47:24.822569 13705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:47:24.806885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52996 (local address 127.28.6.3:42621)
0810 00:47:24.807099 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 00:47:24.807106 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:47:24.807127 (+ 21us) server_negotiation.cc:407] Connection header received
0810 00:47:24.807350 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:47:24.807355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:47:24.807439 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:47:24.807581 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:47:24.808963 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:47:24.809768 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:47:24.811101 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:47:24.811644 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:47:24.811826 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:47:24.816557 (+ 4731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:47:24.816570 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:47:24.816584 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:47:24.816619 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:47:24.819701 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:47:24.820400 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:47:24.820408 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:47:24.820410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:47:24.820553 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:47:24.820987 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:47:24.820991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:47:24.820993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:47:24.821366 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:47:24.821744 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0810 00:47:24.821996 (+ 252us) server_negotiation.cc:299] Negotiation successful
0810 00:47:24.822246 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260810 00:47:24.823483 13673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:47:24.946116 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 13498
I20260810 00:47:24.974654 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 12921
I20260810 00:47:24.975991 13017 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:47:25.090556 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 12921
I20260810 00:47:25.124210 28696 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 00:47:25.459159 28696 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 00:47:25.459362 28696 mini_ranger.cc:66] Stopping Ranger...
I20260810 00:47:25.805204 28696 mini_ranger.cc:68] Stopped Ranger
2026-08-10 00:47:25.805 UTC [12309] LOG: received smart shutdown request
2026-08-10 00:47:25.809 UTC [12309] LOG: background worker "logical replication launcher" (PID 12319) exited with exit code 1
2026-08-10 00:47:25.809 UTC [12314] LOG: shutting down
2026-08-10 00:47:25.965 UTC [12309] LOG: database system is shut down
2026-08-10T00:47:25Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (52267 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:47:26 dist-test-slave-1nl6 krb5kdc[13762](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:47:26 dist-test-slave-1nl6 krb5kdc[13762](info): set up 2 sockets
Aug 10 00:47:26 dist-test-slave-1nl6 krb5kdc[13762](info): commencing operation
krb5kdc: starting...
W20260810 00:47:27.421232 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.280s user 0.004s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:47:27 dist-test-slave-1nl6 krb5kdc[13762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:47:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:47:27Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.6.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.6.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.6.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/rangeradmin_127.28.6.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.6.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/rangeradmin_127.28.6.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.6.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.6.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.6.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/rangerlookup_127.28.6.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.6.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/rangerlookup_127.28.6.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.6.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.6.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.6.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/HTTP_127.28.6.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.6.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/HTTP_127.28.6.20@KRBTEST.COM.keytab.
I20260810 00:47:27.769563 28696 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasksi0adg/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/postgres -l logfile start
2026-08-10 00:47:30.119 UTC [13808] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 00:47:30.119 UTC [13808] LOG: listening on IPv4 address "127.28.6.20", port 56695
2026-08-10 00:47:30.122 UTC [13808] LOG: listening on Unix socket "/tmp/.s.PGSQL.56695"
2026-08-10 00:47:30.147 UTC [13812] LOG: database system was shut down at 2026-08-10 00:47:28 UTC
2026-08-10 00:47:30.151 UTC [13808] LOG: database system is ready to accept connections
I20260810 00:47:31.325824 28696 mini_postgres.cc:96] Postgres bound to 56695
2026-08-10 00:47:31.347 UTC [13824] FATAL: database "slave" does not exist
127.28.6.20:56695 - accepting connections
I20260810 00:47:31.348337 28696 mini_ranger.cc:161] Starting Ranger...
I20260810 00:47:31.376078 28696 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 00:47:31.835237 28696 mini_ranger.cc:87] Created ranger Postgres database
I20260810 00:47:31.835353 28696 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ranger-admin
2026-08-10 00:47:32,122 [I] DB FLAVOR :POSTGRES
2026-08-10 00:47:32,123 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 00:47:32,123 [I] Checking connection..
2026-08-10 00:47:32,123 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:56695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 00:47:32,388 [I] Checking connection passed.
2026-08-10 00:47:32,389 [I] --------- Verifying version history table ---------
2026-08-10 00:47:32,389 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:56695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 00:47:32,639 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 00:47:32,639 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:56695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 00:47:32,906 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 00:47:32,906 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 00:47:32,906 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:56695/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 00:47:33.154 UTC [13909] WARNING: there is no transaction in progress
2026-08-10 00:47:33,163 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 00:47:33,163 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:56695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 00:47:33,422 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 00:47:33,422 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 00:47:33,422 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:56695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 00:47:33,679 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:56695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 00:47:33,932 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:56695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-1nl6.c.gcp-upstream.internal','N') ;"
2026-08-10 00:47:34,184 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:56695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 00:47:34,442 [I] Table x_portal_user does not exist in database ranger
2026-08-10 00:47:34,442 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:56695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 00:47:34,686 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 00:47:34,687 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:56695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 00:47:34,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:56695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 00:47:35,173 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 00:47:35,173 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:56695/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 00:47:36.084 UTC [14082] WARNING: there is no transaction in progress
2026-08-10 00:47:36.100 UTC [14082] WARNING: there is no transaction in progress
2026-08-10 00:47:36.116 UTC [14082] WARNING: there is no transaction in progress
2026-08-10 00:47:36.130 UTC [14082] WARNING: there is no transaction in progress
2026-08-10 00:47:36.149 UTC [14082] WARNING: there is no transaction in progress
2026-08-10 00:47:36.168 UTC [14082] WARNING: there is no transaction in progress
2026-08-10 00:47:36.229 UTC [14082] WARNING: there is no transaction in progress
2026-08-10 00:47:36.236 UTC [14082] WARNING: there is no transaction in progress
2026-08-10 00:47:36.250 UTC [14082] WARNING: there is no transaction in progress
2026-08-10 00:47:36.260 UTC [14082] WARNING: there is no transaction in progress
2026-08-10 00:47:36.269 UTC [14082] WARNING: there is no transaction in progress
2026-08-10 00:47:36.282 UTC [14082] WARNING: there is no transaction in progress
2026-08-10 00:47:36.291 UTC [14082] WARNING: there is no transaction in progress
2026-08-10 00:47:36.296 UTC [14082] WARNING: there is no transaction in progress
2026-08-10 00:47:36.302 UTC [14082] WARNING: there is no transaction in progress
2026-08-10 00:47:36.308 UTC [14082] WARNING: there is no transaction in progress
2026-08-10 00:47:36,885 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 00:47:36,885 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:56695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 00:47:37,136 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 00:47:37,137 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:56695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 00:47:37,391 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 00:47:37,391 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:56695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 00:47:37,648 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 00:47:37,648 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:56695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 00:47:37,883 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:56695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 00:47:38,130 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:56695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-1nl6.c.gcp-upstream.internal';"
2026-08-10 00:47:38,361 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 00:47:38,361 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:56695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 00:47:38,591 [I] DB_PATCHES have already been applied
I20260810 00:47:38.597579 28696 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ranger-admin:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasksi0adg/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 00:47:38.597666 28696 mini_ranger.cc:193] Using host: 127.28.6.20
I20260810 00:47:38.617981 28696 mini_ranger.cc:227] Ranger admin URL: http://127.28.6.20:51315
Aug 10, 2026 12:47:38 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 12:47:38 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 12:47:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 12:47:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 12:47:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 12:47:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 12:47:39 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 00:47:40.010561 28696 mini_ranger.cc:160] Time spent starting Ranger: real 8.662s user 0.001s sys 0.052s
Aug 10, 2026 12:48:00 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 12:48:00 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 12:48:00 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 12:48:00 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 12:48:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 12:48:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 12:48:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 12:48:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 12:48:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 12:48:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 12:48:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 12:48:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 12:48:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 12:48:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 12:48:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 12:48:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 12:48:00 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 12:48:01 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 00:48:01.792165 28696 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.6.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.6.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.6.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/rangerkms_127.28.6.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.6.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/rangerkms_127.28.6.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.6.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.6.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.6.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/HTTP_127.28.6.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.6.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/HTTP_127.28.6.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 00:48:01 dist-test-slave-1nl6 krb5kdc[13762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322881, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 00:48:01.982963 28696 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 00:48:02.006815 28696 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 00:48:02.733175 28696 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 00:48:02.733309 28696 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ranger-kms
I20260810 00:48:02.733382 28696 mini_ranger_kms.cc:227] Using postgres at 127.28.6.20:56695
2026-08-10 00:48:03,112 [I] DB FLAVOR :POSTGRES
2026-08-10 00:48:03,112 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 00:48:03,112 [I] Checking connection
2026-08-10 00:48:03,112 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:56695/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 00:48:03,371 [I] connection success
2026-08-10 00:48:03,371 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 00:48:03,372 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 00:48:03,372 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:56695/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 00:48:03,607 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 00:48:03,607 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 00:48:03,607 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 00:48:03,607 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.6.20:56695/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 00:48:03,878 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 00:48:04.367985 28696 mini_ranger_kms.cc:310] Created kms service
I20260810 00:48:04.556183 28696 mini_ranger_kms.cc:326] Created kudu user
I20260810 00:48:04.600575 28696 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 00:48:04 dist-test-slave-1nl6 krb5kdc[13762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322881, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.6.20@KRBTEST.COM
I20260810 00:48:04.969981 28696 mini_ranger_kms.cc:384] Added ranger policy
I20260810 00:48:04.970293 28696 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ranger-kms:/tmp/dist-test-tasksi0adg/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasksi0adg/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 00:48:04.970358 28696 mini_ranger_kms.cc:240] Using host: 127.28.6.20
I20260810 00:48:04.996132 28696 mini_ranger_kms.cc:276] Ranger KMS PID: 14352
I20260810 00:48:04.996277 28696 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.6.20:46067
Aug 10, 2026 12:48:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 12:48:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 12:48:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 12:48:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 12:48:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 12:48:06 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 00:48:07 dist-test-slave-1nl6 krb5kdc[13762](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786322887, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.6.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 10 00:48:07 dist-test-slave-1nl6 krb5kdc[13762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322887, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.6.20@KRBTEST.COM for HTTP/127.28.6.20@KRBTEST.COM
I20260810 00:48:07.765571 28696 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.783s user 0.085s sys 0.321s
I20260810 00:48:07.765818 28696 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 00:48:07.765978 28696 mini_ranger_kms.cc:401] 127.28.6.20:46067/kms/v1/keys
Aug 10 00:48:08 dist-test-slave-1nl6 krb5kdc[13762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322887, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.6.20@KRBTEST.COM for HTTP/127.28.6.20@KRBTEST.COM
Aug 10 00:48:10 dist-test-slave-1nl6 krb5kdc[13762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.6.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
I20260810 00:48:10.477212 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:33563
--webserver_interface=127.28.6.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:43281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:33563
--ranger_config_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.6.20:46067/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:48:10.827212 14430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:48:10.827481 14430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:48:10.827641 14430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:48:10.834630 14430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:48:10.834722 14430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:48:10.834836 14430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:48:10.834940 14430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:48:10.835029 14430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:48:10.844617 14430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:43281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.6.20:46067/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:33563
--ranger_config_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:33563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:48:10.845750 14430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:48:10.847098 14430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:48:10.859221 14440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:10.859349 14436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:10.859310 14438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:48:10.859855 14430 server_base.cc:1034] running on GCE node
I20260810 00:48:10.860633 14430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:48:10.861950 14430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:48:10.863160 14430 hybrid_clock.cc:648] HybridClock initialized: now 1786322890863107 us; error 34 us; skew 500 ppm
I20260810 00:48:10.863565 14430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:48:10 dist-test-slave-1nl6 krb5kdc[13762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322890, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:48:10.866537 14430 init.cc:377] Logged in from keytab as kudu/127.28.6.62@KRBTEST.COM (short username kudu)
I20260810 00:48:10.867954 14430 webserver.cc:460] Webserver started at http://127.28.6.62:44813/ using document root <none> and password file <none>
I20260810 00:48:10.868531 14430 fs_manager.cc:359] Metadata directory not provided
I20260810 00:48:10.868630 14430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:48:10.868878 14430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 00:48:10 dist-test-slave-1nl6 krb5kdc[13762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322890, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for HTTP/127.28.6.20@KRBTEST.COM
I20260810 00:48:10.925521 14430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "72da27ef4e3d40da825620568c4face5"
format_stamp: "Formatted at 2026-08-10 00:48:10 on dist-test-slave-1nl6"
server_key: "8239b3d2149877a7223800fe8acba99e"
server_key_iv: "c452c1d424651d106be91cebe3cb5dc0"
server_key_version: "kuduclusterkey@0"
I20260810 00:48:10.926347 14430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "72da27ef4e3d40da825620568c4face5"
format_stamp: "Formatted at 2026-08-10 00:48:10 on dist-test-slave-1nl6"
server_key: "8239b3d2149877a7223800fe8acba99e"
server_key_iv: "c452c1d424651d106be91cebe3cb5dc0"
server_key_version: "kuduclusterkey@0"
I20260810 00:48:10.931505 14430 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 00:48:10.951478 14451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:10.953035 14430 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260810 00:48:10.953244 14430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "72da27ef4e3d40da825620568c4face5"
format_stamp: "Formatted at 2026-08-10 00:48:10 on dist-test-slave-1nl6"
server_key: "8239b3d2149877a7223800fe8acba99e"
server_key_iv: "c452c1d424651d106be91cebe3cb5dc0"
server_key_version: "kuduclusterkey@0"
I20260810 00:48:10.953486 14430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:48:10.971448 14430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:48:10.975102 14430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:48:10.975427 14430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:48:10.994189 14430 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:33563
I20260810 00:48:10.994226 14537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:33563 every 8 connection(s)
I20260810 00:48:10.995673 14430 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:48:10.999505 14539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:48:11.005569 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 14430
I20260810 00:48:11.005801 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:48:11.006723 28696 external_mini_cluster.cc:1428] Setting key a81399f83eb25d8d08122ad4a0e183b4
I20260810 00:48:11.010812 14539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5: Bootstrap starting.
I20260810 00:48:11.014542 14539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5: Neither blocks nor log segments found. Creating new log.
I20260810 00:48:11.015704 14539 log.cc:824] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5: Log is configured to *not* fsync() on all Append() calls
I20260810 00:48:11.018837 14539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5: No bootstrap required, opened a new log
Aug 10 00:48:11 dist-test-slave-1nl6 krb5kdc[13762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:48:11.026845 14539 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72da27ef4e3d40da825620568c4face5" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 33563 } }
I20260810 00:48:11.027307 14539 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:48:11.027432 14539 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72da27ef4e3d40da825620568c4face5, State: Initialized, Role: FOLLOWER
I20260810 00:48:11.028000 14539 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72da27ef4e3d40da825620568c4face5" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 33563 } }
I20260810 00:48:11.028242 14539 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:48:11.028411 14539 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:48:11.028616 14539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:48:11.030514 14543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:11.008921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49604 (local address 127.28.6.62:33563)
0810 00:48:11.009741 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:11.009753 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:11.009795 (+ 42us) server_negotiation.cc:407] Connection header received
0810 00:48:11.011579 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:11.011603 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:11.011984 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:11.012340 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:11.014089 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:11.015507 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:11.017148 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:11.017787 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:11.017929 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:11.022428 (+ 4499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:11.022436 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:11.022450 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:11.022495 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:11.025785 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:11.026502 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:11.026513 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:11.026525 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:11.026666 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:11.027147 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:11.027155 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:11.027162 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:11.027912 (+ 750us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:11.028242 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:11.028700 (+ 458us) server_negotiation.cc:299] Negotiation successful
0810 00:48:11.029136 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":399,"threads_started":1}
I20260810 00:48:11.031646 14539 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72da27ef4e3d40da825620568c4face5" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 33563 } }
I20260810 00:48:11.032140 14539 leader_election.cc:302] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72da27ef4e3d40da825620568c4face5; no voters:
I20260810 00:48:11.032857 14539 leader_election.cc:288] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:48:11.032965 14547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:48:11.033911 14547 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5 [term 1 LEADER]: Becoming Leader. State: Replica: 72da27ef4e3d40da825620568c4face5, State: Running, Role: LEADER
I20260810 00:48:11.034507 14547 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72da27ef4e3d40da825620568c4face5" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 33563 } }
I20260810 00:48:11.035493 14539 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:48:11.037467 14551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72da27ef4e3d40da825620568c4face5. Latest consensus state: current_term: 1 leader_uuid: "72da27ef4e3d40da825620568c4face5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72da27ef4e3d40da825620568c4face5" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 33563 } } }
I20260810 00:48:11.037572 14550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72da27ef4e3d40da825620568c4face5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72da27ef4e3d40da825620568c4face5" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 33563 } } }
I20260810 00:48:11.037926 14551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5 [sys.catalog]: This master's current role is: LEADER
I20260810 00:48:11.037961 14550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5 [sys.catalog]: This master's current role is: LEADER
I20260810 00:48:11.043073 14554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:48:11.048784 14554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:48:11.049413 14539 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 00:48:11.070668 14554 catalog_manager.cc:1347] Generated new cluster ID: 26b6a889c71544a8a3ab6cd1331767c7
I20260810 00:48:11.071002 14554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:48:11.088274 14554 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:48:11.090337 14554 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:48:11.098985 14554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5: Generated new TSK 0
I20260810 00:48:11.100158 14554 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.6.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab.
Entry for principal kudu/127.28.6.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab.
I20260810 00:48:11.862362 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.6.20:46067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.1:0
--local_ip_for_outbound_sockets=127.28.6.1
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:33563
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:43281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 00:48:12 dist-test-slave-1nl6 krb5kdc[13762](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786322892, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 00:48:12.273605 14602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:48:12.273937 14602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:48:12.274111 14602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:48:12.282369 14602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:48:12.282560 14602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:48:12.282804 14602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.1
I20260810 00:48:12.294539 14602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:43281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.6.20:46067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:33563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.1
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:48:12.296048 14602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:48:12.297871 14602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:48:12.315788 14624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:12.316401 14629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:48:12.324924 14602 server_base.cc:1034] running on GCE node
W20260810 00:48:12.325555 14631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:48:12.326256 14602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:48:12.327543 14602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:48:12.328830 14602 hybrid_clock.cc:648] HybridClock initialized: now 1786322892328743 us; error 77 us; skew 500 ppm
I20260810 00:48:12.329201 14602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:48:12 dist-test-slave-1nl6 krb5kdc[13762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322892, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:48:12.332599 14602 init.cc:377] Logged in from keytab as kudu/127.28.6.1@KRBTEST.COM (short username kudu)
I20260810 00:48:12.334509 14602 webserver.cc:460] Webserver started at http://127.28.6.1:36253/ using document root <none> and password file <none>
I20260810 00:48:12.335356 14602 fs_manager.cc:359] Metadata directory not provided
I20260810 00:48:12.335506 14602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:48:12.335855 14602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 00:48:12 dist-test-slave-1nl6 krb5kdc[13762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322892, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for HTTP/127.28.6.20@KRBTEST.COM
I20260810 00:48:12.353832 14602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-0/data/instance:
uuid: "aea2826c1e1d4742801c756244bcaf2c"
format_stamp: "Formatted at 2026-08-10 00:48:12 on dist-test-slave-1nl6"
server_key: "4bffa9affe67f2ed29e4177c894b59c0"
server_key_iv: "a6a555fc80855515caf1f83c952ba490"
server_key_version: "kuduclusterkey@0"
I20260810 00:48:12.356827 14602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance:
uuid: "aea2826c1e1d4742801c756244bcaf2c"
format_stamp: "Formatted at 2026-08-10 00:48:12 on dist-test-slave-1nl6"
server_key: "4bffa9affe67f2ed29e4177c894b59c0"
server_key_iv: "a6a555fc80855515caf1f83c952ba490"
server_key_version: "kuduclusterkey@0"
I20260810 00:48:12.371308 14602 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.005s sys 0.008s
I20260810 00:48:12.400200 14645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:12.402061 14602 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.004s sys 0.002s
I20260810 00:48:12.402313 14602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-0/wal
uuid: "aea2826c1e1d4742801c756244bcaf2c"
format_stamp: "Formatted at 2026-08-10 00:48:12 on dist-test-slave-1nl6"
server_key: "4bffa9affe67f2ed29e4177c894b59c0"
server_key_iv: "a6a555fc80855515caf1f83c952ba490"
server_key_version: "kuduclusterkey@0"
I20260810 00:48:12.402639 14602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:48:12.432914 14602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:48:12.438190 14602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:48:12.438583 14602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:48:12.445925 14602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:48:12.452554 14602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:48:12.452661 14602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:12.452907 14602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:48:12.453048 14602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:12.567691 14602 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.1:36715
I20260810 00:48:12.567715 14809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.1:36715 every 8 connection(s)
I20260810 00:48:12.569447 14602 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
I20260810 00:48:12.578616 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 14602
I20260810 00:48:12.578827 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance
I20260810 00:48:12.579643 28696 external_mini_cluster.cc:1428] Setting key 61d58385d44dd8c703ce3d56a36173ea
Aug 10 00:48:12 dist-test-slave-1nl6 krb5kdc[13762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322892, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:48:12.625941 14813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:12.575836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:60499 (local address 127.28.6.62:33563)
0810 00:48:12.598238 (+ 22402us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:12.598249 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:12.606773 (+ 8524us) server_negotiation.cc:407] Connection header received
0810 00:48:12.606844 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:12.606850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:12.606985 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:12.607094 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:12.609286 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:12.610485 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:12.612952 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:12.613577 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:12.613673 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:12.618311 (+ 4638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:12.618318 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:12.618322 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:12.618361 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:12.622120 (+ 3759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:12.623085 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:12.623091 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:12.623094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:12.623250 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:12.623828 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:12.623832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:12.623835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:12.624208 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:12.624336 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:12.625298 (+ 962us) server_negotiation.cc:299] Negotiation successful
0810 00:48:12.625533 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20979,"thread_start_us":20845,"threads_started":1}
I20260810 00:48:12.628482 14814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:12.576466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:33563 (local address 127.28.6.1:60499)
0810 00:48:12.603902 (+ 27436us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:12.603962 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:12.606395 (+ 2433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:12.607547 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:12.607561 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:12.608222 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:48:12.609079 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:12.609106 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:12.610704 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:12.610710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:12.612789 (+ 2079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:12.612804 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:12.614020 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:12.614027 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:12.614322 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:12.615190 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:48:12.615221 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:48:12.618040 (+ 2819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:48:12.622308 (+ 4268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:12.622316 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:12.622334 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:12.622946 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:12.623403 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:12.623408 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:12.623411 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:12.623673 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:12.624441 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:48:12.624449 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:48:12.624855 (+ 406us) client_negotiation.cc:769] Sending connection context
0810 00:48:12.626466 (+ 1611us) client_negotiation.cc:240] Negotiation successful
0810 00:48:12.627040 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27103,"thread_start_us":26513,"threads_started":1}
I20260810 00:48:12.630651 14810 heartbeater.cc:347] Connected to a master server at 127.28.6.62:33563
I20260810 00:48:12.631050 14810 heartbeater.cc:464] Registering TS with master...
I20260810 00:48:12.631980 14810 heartbeater.cc:511] Master 127.28.6.62:33563 requested a full tablet report, sending...
I20260810 00:48:12.634645 14481 ts_manager.cc:194] Registered new tserver with Master: aea2826c1e1d4742801c756244bcaf2c (127.28.6.1:36715)
I20260810 00:48:12.636816 14481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.1@KRBTEST.COM'} at 127.28.6.1:60499
WARNING: no policy specified for kudu/127.28.6.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.2@KRBTEST.COM" created.
Aug 10 00:48:12 dist-test-slave-1nl6 krb5kdc[13762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322892, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for HTTP/127.28.6.20@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab.
Entry for principal kudu/127.28.6.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab.
W20260810 00:48:12.689425 28696 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.28.6.2: real 0.107s user 0.010s sys 0.015s
I20260810 00:48:12.690799 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.6.20:46067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.2:0
--local_ip_for_outbound_sockets=127.28.6.2
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:33563
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:43281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 00:48:12 dist-test-slave-1nl6 krb5kdc[13762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322892, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for HTTP/127.28.6.20@KRBTEST.COM
W20260810 00:48:13.204583 14817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:48:13.204952 14817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:48:13.205127 14817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:48:13.216361 14817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:48:13.216545 14817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:48:13.216759 14817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.2
I20260810 00:48:13.229547 14817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:43281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.6.20:46067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:33563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.2
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:48:13.231343 14817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:48:13.233165 14817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:48:13.252344 14833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:13.252310 14836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:13.253079 14845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:48:13.253444 14817 server_base.cc:1034] running on GCE node
I20260810 00:48:13.254102 14817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:48:13.255116 14817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:48:13.256430 14817 hybrid_clock.cc:648] HybridClock initialized: now 1786322893256364 us; error 57 us; skew 500 ppm
I20260810 00:48:13.256786 14817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:48:13 dist-test-slave-1nl6 krb5kdc[13762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322893, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:48:13.260290 14817 init.cc:377] Logged in from keytab as kudu/127.28.6.2@KRBTEST.COM (short username kudu)
I20260810 00:48:13.261850 14817 webserver.cc:460] Webserver started at http://127.28.6.2:46603/ using document root <none> and password file <none>
I20260810 00:48:13.262609 14817 fs_manager.cc:359] Metadata directory not provided
I20260810 00:48:13.262753 14817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:48:13.263136 14817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 00:48:13 dist-test-slave-1nl6 krb5kdc[13762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322893, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for HTTP/127.28.6.20@KRBTEST.COM
I20260810 00:48:13.278488 14817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-1/data/instance:
uuid: "1f0de896a4134165b1ab0c26f803f26b"
format_stamp: "Formatted at 2026-08-10 00:48:13 on dist-test-slave-1nl6"
server_key: "7fdcf2d61adc7166f9efb945ca8d8cb2"
server_key_iv: "23d8f33b3e2d4df233b194d87f8feeda"
server_key_version: "kuduclusterkey@0"
I20260810 00:48:13.279340 14817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance:
uuid: "1f0de896a4134165b1ab0c26f803f26b"
format_stamp: "Formatted at 2026-08-10 00:48:13 on dist-test-slave-1nl6"
server_key: "7fdcf2d61adc7166f9efb945ca8d8cb2"
server_key_iv: "23d8f33b3e2d4df233b194d87f8feeda"
server_key_version: "kuduclusterkey@0"
I20260810 00:48:13.284750 14817 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 00:48:13.299686 14854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:13.301302 14817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 00:48:13.301497 14817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-1/wal
uuid: "1f0de896a4134165b1ab0c26f803f26b"
format_stamp: "Formatted at 2026-08-10 00:48:13 on dist-test-slave-1nl6"
server_key: "7fdcf2d61adc7166f9efb945ca8d8cb2"
server_key_iv: "23d8f33b3e2d4df233b194d87f8feeda"
server_key_version: "kuduclusterkey@0"
I20260810 00:48:13.301729 14817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:48:13.327118 14817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:48:13.331715 14817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:48:13.332039 14817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:48:13.333151 14817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:48:13.334817 14817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:48:13.334932 14817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:13.335053 14817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:48:13.335115 14817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:13.377641 14817 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.2:33139
I20260810 00:48:13.377661 15048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.2:33139 every 8 connection(s)
I20260810 00:48:13.379294 14817 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
I20260810 00:48:13.384434 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 14817
I20260810 00:48:13.384621 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance
I20260810 00:48:13.385175 28696 external_mini_cluster.cc:1428] Setting key 55f6d8fc30f65b4cd3c5936fe0a7a698
Aug 10 00:48:13 dist-test-slave-1nl6 krb5kdc[13762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322893, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:48:13.406822 15055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:13.381998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:39857 (local address 127.28.6.62:33563)
0810 00:48:13.382511 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:13.382518 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:13.383621 (+ 1103us) server_negotiation.cc:407] Connection header received
0810 00:48:13.385626 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:13.385632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:13.385754 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:13.385908 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:13.391013 (+ 5105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:13.392282 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:13.394628 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:13.395358 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:13.395506 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:13.399686 (+ 4180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:13.399692 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:13.399695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:13.399726 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:13.402966 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:13.403966 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:13.403972 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:13.403976 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:13.404139 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:13.404696 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:13.404701 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:13.404705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:13.405063 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:13.405202 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:13.406125 (+ 923us) server_negotiation.cc:299] Negotiation successful
0810 00:48:13.406370 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":251,"threads_started":1}
I20260810 00:48:13.408367 15056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:13.382503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:33563 (local address 127.28.6.2:39857)
0810 00:48:13.383420 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:13.383496 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:13.385361 (+ 1865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:13.386176 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:13.386188 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:13.386808 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:48:13.387635 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:13.387661 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:13.392469 (+ 4808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:13.392475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:13.394459 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:13.394475 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:13.395610 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:13.395617 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:13.395900 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:13.396692 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:48:13.396727 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:48:13.399474 (+ 2747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:48:13.403115 (+ 3641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:13.403127 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:13.403146 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:13.403812 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:13.404276 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:13.404283 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:13.404290 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:13.404554 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:13.405218 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:48:13.405230 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:48:13.405647 (+ 417us) client_negotiation.cc:769] Sending connection context
0810 00:48:13.405977 (+ 330us) client_negotiation.cc:240] Negotiation successful
0810 00:48:13.406523 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":414,"threads_started":1}
I20260810 00:48:13.409993 15049 heartbeater.cc:347] Connected to a master server at 127.28.6.62:33563
I20260810 00:48:13.410306 15049 heartbeater.cc:464] Registering TS with master...
I20260810 00:48:13.411175 15049 heartbeater.cc:511] Master 127.28.6.62:33563 requested a full tablet report, sending...
I20260810 00:48:13.412984 14481 ts_manager.cc:194] Registered new tserver with Master: 1f0de896a4134165b1ab0c26f803f26b (127.28.6.2:33139)
I20260810 00:48:13.414098 14481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.2@KRBTEST.COM'} at 127.28.6.2:39857
WARNING: no policy specified for kudu/127.28.6.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab.
Entry for principal kudu/127.28.6.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab.
I20260810 00:48:13.450474 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.6.20:46067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.3:0
--local_ip_for_outbound_sockets=127.28.6.3
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:33563
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:43281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 00:48:13.640686 14810 heartbeater.cc:503] Master 127.28.6.62:33563 was elected leader, sending a full tablet report...
W20260810 00:48:13.780371 15061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:48:13.780629 15061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:48:13.780793 15061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:48:13.787750 15061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:48:13.787842 15061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:48:13.788002 15061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.3
I20260810 00:48:13.797420 15061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:43281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.6.20:46067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:33563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.3
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:48:13.798578 15061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:48:13.800060 15061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:48:13.812616 15070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:13.812597 15071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:13.812837 15076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:48:13.813688 15061 server_base.cc:1034] running on GCE node
I20260810 00:48:13.814291 15061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:48:13.815213 15061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:48:13.816440 15061 hybrid_clock.cc:648] HybridClock initialized: now 1786322893816404 us; error 45 us; skew 500 ppm
I20260810 00:48:13.816744 15061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:48:13 dist-test-slave-1nl6 krb5kdc[13762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322893, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:48:13.819885 15061 init.cc:377] Logged in from keytab as kudu/127.28.6.3@KRBTEST.COM (short username kudu)
I20260810 00:48:13.821309 15061 webserver.cc:460] Webserver started at http://127.28.6.3:46839/ using document root <none> and password file <none>
I20260810 00:48:13.821947 15061 fs_manager.cc:359] Metadata directory not provided
I20260810 00:48:13.822091 15061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:48:13.822400 15061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 00:48:13 dist-test-slave-1nl6 krb5kdc[13762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322893, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for HTTP/127.28.6.20@KRBTEST.COM
I20260810 00:48:13.838495 15061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-2/data/instance:
uuid: "30d3f8306f584df2b93842c33e43bfb0"
format_stamp: "Formatted at 2026-08-10 00:48:13 on dist-test-slave-1nl6"
server_key: "0e55d7b33ad034b6186bfd5a97d6dfde"
server_key_iv: "6ad8df0ccc3452990f68203d29eae85c"
server_key_version: "kuduclusterkey@0"
I20260810 00:48:13.839686 15061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance:
uuid: "30d3f8306f584df2b93842c33e43bfb0"
format_stamp: "Formatted at 2026-08-10 00:48:13 on dist-test-slave-1nl6"
server_key: "0e55d7b33ad034b6186bfd5a97d6dfde"
server_key_iv: "6ad8df0ccc3452990f68203d29eae85c"
server_key_version: "kuduclusterkey@0"
I20260810 00:48:13.846757 15061 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260810 00:48:13.860721 15084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:13.862056 15061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 00:48:13.862242 15061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-2/wal
uuid: "30d3f8306f584df2b93842c33e43bfb0"
format_stamp: "Formatted at 2026-08-10 00:48:13 on dist-test-slave-1nl6"
server_key: "0e55d7b33ad034b6186bfd5a97d6dfde"
server_key_iv: "6ad8df0ccc3452990f68203d29eae85c"
server_key_version: "kuduclusterkey@0"
I20260810 00:48:13.862481 15061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:48:13.883322 15061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:48:13.886955 15061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:48:13.887293 15061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:48:13.888257 15061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:48:13.889582 15061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:48:13.889657 15061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:13.889839 15061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:48:13.889923 15061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:13.924449 15061 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.3:46159
I20260810 00:48:13.924495 15252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.3:46159 every 8 connection(s)
I20260810 00:48:13.925755 15061 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
I20260810 00:48:13.934429 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 15061
I20260810 00:48:13.934621 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance
I20260810 00:48:13.935261 28696 external_mini_cluster.cc:1428] Setting key 247ffd9910fa1e9c3241d770bdfcf5f4
Aug 10 00:48:13 dist-test-slave-1nl6 krb5kdc[13762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322893, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:48:13.945865 15257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:13.928335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:44473 (local address 127.28.6.62:33563)
0810 00:48:13.928834 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:13.928838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:13.929803 (+ 965us) server_negotiation.cc:407] Connection header received
0810 00:48:13.931727 (+ 1924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:13.931732 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:13.931836 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:13.931958 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:13.933406 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:13.934143 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:13.936029 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:13.936710 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:13.936850 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:13.940211 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:13.940216 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:13.940219 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:13.940252 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:13.942705 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:13.943491 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:13.943497 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:13.943508 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:13.943636 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:13.944080 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:13.944084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:13.944086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:13.944415 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:13.944571 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:13.945253 (+ 682us) server_negotiation.cc:299] Negotiation successful
0810 00:48:13.945485 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":240,"threads_started":1}
I20260810 00:48:13.946806 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:13.928905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:33563 (local address 127.28.6.3:44473)
0810 00:48:13.929627 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:13.929673 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:13.931433 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:13.932206 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:13.932213 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:13.932651 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:48:13.933218 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:13.933236 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:13.934322 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:13.934326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:13.935878 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:13.935891 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:13.936943 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:13.936950 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:13.937195 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:13.937874 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:48:13.937897 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:48:13.939983 (+ 2086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:48:13.942858 (+ 2875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:13.942867 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:13.942884 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:13.943357 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:13.943752 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:13.943757 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:13.943760 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:13.943947 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:13.944545 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:48:13.944553 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:48:13.944885 (+ 332us) client_negotiation.cc:769] Sending connection context
0810 00:48:13.945145 (+ 260us) client_negotiation.cc:240] Negotiation successful
0810 00:48:13.945696 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":312,"threads_started":1}
I20260810 00:48:13.948184 15255 heartbeater.cc:347] Connected to a master server at 127.28.6.62:33563
I20260810 00:48:13.948457 15255 heartbeater.cc:464] Registering TS with master...
I20260810 00:48:13.949177 15255 heartbeater.cc:511] Master 127.28.6.62:33563 requested a full tablet report, sending...
I20260810 00:48:13.950891 14481 ts_manager.cc:194] Registered new tserver with Master: 30d3f8306f584df2b93842c33e43bfb0 (127.28.6.3:46159)
I20260810 00:48:13.951748 14481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.3@KRBTEST.COM'} at 127.28.6.3:44473
I20260810 00:48:13.962590 28696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 00:48:13.964323 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 14602
Aug 10 00:48:13 dist-test-slave-1nl6 krb5kdc[13762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.1@KRBTEST.COM
I20260810 00:48:13.981559 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:13.965231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59014 (local address 127.28.6.1:36715)
0810 00:48:13.965951 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:13.965958 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:13.965982 (+ 24us) server_negotiation.cc:407] Connection header received
0810 00:48:13.966049 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:13.966055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:13.966231 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:13.966388 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:13.967855 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:13.968844 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:13.970352 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:13.970887 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:13.971077 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:13.975648 (+ 4571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:13.975659 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:13.975687 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:13.975723 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:13.978670 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:13.979706 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:13.979713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:13.979715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:13.979873 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:13.980188 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:13.980191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:13.980193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:13.980504 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:13.980724 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:13.980958 (+ 234us) server_negotiation.cc:299] Negotiation successful
0810 00:48:13.981148 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":336,"threads_started":1}
I20260810 00:48:13.982838 14804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:48:14.065963 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 14602
I20260810 00:48:14.090516 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 14817
Aug 10 00:48:14 dist-test-slave-1nl6 krb5kdc[13762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.2@KRBTEST.COM
I20260810 00:48:14.111760 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:14.096893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51424 (local address 127.28.6.2:33139)
0810 00:48:14.097489 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:14.097496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:14.097522 (+ 26us) server_negotiation.cc:407] Connection header received
0810 00:48:14.097600 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:14.097606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:14.097817 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:14.097954 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:14.099234 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:14.100340 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:14.101560 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:14.102088 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:14.102184 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:14.106382 (+ 4198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:14.106394 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:14.106408 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:14.106440 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:14.109341 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:14.109969 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:14.109975 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:14.109977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:14.110100 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:14.110412 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:14.110415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:14.110416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:14.110718 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:14.111025 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:14.111265 (+ 240us) server_negotiation.cc:299] Negotiation successful
0810 00:48:14.111458 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":293,"threads_started":1}
I20260810 00:48:14.113085 15038 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:48:14.201543 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 14817
I20260810 00:48:14.227650 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 15061
Aug 10 00:48:14 dist-test-slave-1nl6 krb5kdc[13762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.3@KRBTEST.COM
I20260810 00:48:14.248240 15266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:14.228694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47708 (local address 127.28.6.3:46159)
0810 00:48:14.229350 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:14.229357 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:14.229387 (+ 30us) server_negotiation.cc:407] Connection header received
0810 00:48:14.229449 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:14.229455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:14.229669 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:14.229845 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:14.231273 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:14.232299 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:14.233777 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:14.234340 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:14.234495 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:14.242316 (+ 7821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:14.242327 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:14.242342 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:14.242375 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:14.245369 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:14.246046 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:14.246054 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:14.246057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:14.246185 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:14.246591 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:14.246596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:14.246601 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:14.246993 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:14.247343 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:14.247623 (+ 280us) server_negotiation.cc:299] Negotiation successful
0810 00:48:14.247885 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":337,"threads_started":1}
I20260810 00:48:14.249465 15245 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:48:14.330667 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 15061
I20260810 00:48:14.355605 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 14430
I20260810 00:48:14.356936 14530 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:48:14.462807 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 14430
I20260810 00:48:14.497941 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:33563
--webserver_interface=127.28.6.62
--webserver_port=44813
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:43281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:33563
--ranger_config_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.6.20:46067/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.6.20:46067/kms with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:48:14.823485 15269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:48:14.823738 15269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:48:14.823822 15269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:48:14.831122 15269 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:48:14.831257 15269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:48:14.831311 15269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:48:14.831362 15269 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:48:14.831403 15269 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:48:14.841096 15269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:43281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.6.20:46067/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:33563
--ranger_config_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:33563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=44813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:48:14.842170 15269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:48:14.843418 15269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:48:14.854969 15278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:14.854995 15284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:14.854928 15281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:48:14.855813 15269 server_base.cc:1034] running on GCE node
I20260810 00:48:14.856477 15269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:48:14.857761 15269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:48:14.858999 15269 hybrid_clock.cc:648] HybridClock initialized: now 1786322894858931 us; error 50 us; skew 500 ppm
I20260810 00:48:14.859303 15269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:48:14 dist-test-slave-1nl6 krb5kdc[13762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322894, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:48:14.862295 15269 init.cc:377] Logged in from keytab as kudu/127.28.6.62@KRBTEST.COM (short username kudu)
I20260810 00:48:14.863754 15269 webserver.cc:460] Webserver started at http://127.28.6.62:44813/ using document root <none> and password file <none>
I20260810 00:48:14.864317 15269 fs_manager.cc:359] Metadata directory not provided
I20260810 00:48:14.864428 15269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 00:48:14 dist-test-slave-1nl6 krb5kdc[13762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322894, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for HTTP/127.28.6.20@KRBTEST.COM
I20260810 00:48:14.951829 15269 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 00:48:14.955009 15298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:14.956391 15269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 00:48:14.956575 15269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "72da27ef4e3d40da825620568c4face5"
format_stamp: "Formatted at 2026-08-10 00:48:10 on dist-test-slave-1nl6"
server_key: "8239b3d2149877a7223800fe8acba99e"
server_key_iv: "c452c1d424651d106be91cebe3cb5dc0"
server_key_version: "kuduclusterkey@0"
I20260810 00:48:14.957692 15269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:48:14.968729 15269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:48:14.972332 15269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:48:14.972654 15269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:48:14.991262 15269 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:33563
I20260810 00:48:14.991294 15371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:33563 every 8 connection(s)
I20260810 00:48:14.992756 15269 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:48:14.993251 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 15269
I20260810 00:48:14.994306 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.6.20:46067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.1:36715
--local_ip_for_outbound_sockets=127.28.6.1
--tserver_master_addrs=127.28.6.62:33563
--webserver_port=36253
--webserver_interface=127.28.6.1
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:43281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.6.20:46067/kms with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 00:48:14.999801 15372 sys_catalog.cc:263] Verifying existing consensus state
I20260810 00:48:15.003146 15372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5: Bootstrap starting.
I20260810 00:48:15.027822 15372 log.cc:824] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5: Log is configured to *not* fsync() on all Append() calls
I20260810 00:48:15.037082 15372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 00:48:15.037648 15372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5: Bootstrap complete.
I20260810 00:48:15.046684 15372 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72da27ef4e3d40da825620568c4face5" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 33563 } }
I20260810 00:48:15.047603 15372 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72da27ef4e3d40da825620568c4face5, State: Initialized, Role: FOLLOWER
I20260810 00:48:15.048313 15372 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72da27ef4e3d40da825620568c4face5" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 33563 } }
I20260810 00:48:15.048607 15372 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:48:15.048810 15372 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:48:15.049032 15372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5 [term 1 FOLLOWER]: Advancing to term 2
I20260810 00:48:15.052927 15372 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72da27ef4e3d40da825620568c4face5" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 33563 } }
I20260810 00:48:15.053411 15372 leader_election.cc:302] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72da27ef4e3d40da825620568c4face5; no voters:
I20260810 00:48:15.054046 15372 leader_election.cc:288] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 00:48:15.054214 15378 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 00:48:15.055459 15378 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5 [term 2 LEADER]: Becoming Leader. State: Replica: 72da27ef4e3d40da825620568c4face5, State: Running, Role: LEADER
I20260810 00:48:15.056111 15378 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72da27ef4e3d40da825620568c4face5" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 33563 } }
I20260810 00:48:15.056371 15372 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:48:15.058645 15381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72da27ef4e3d40da825620568c4face5. Latest consensus state: current_term: 2 leader_uuid: "72da27ef4e3d40da825620568c4face5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72da27ef4e3d40da825620568c4face5" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 33563 } } }
I20260810 00:48:15.059015 15381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5 [sys.catalog]: This master's current role is: LEADER
I20260810 00:48:15.059603 15380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "72da27ef4e3d40da825620568c4face5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72da27ef4e3d40da825620568c4face5" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 33563 } } }
I20260810 00:48:15.059900 15380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5 [sys.catalog]: This master's current role is: LEADER
I20260810 00:48:15.070549 15391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:48:15.076329 15391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:48:15.077754 15391 catalog_manager.cc:1259] Loaded cluster ID: 26b6a889c71544a8a3ab6cd1331767c7
I20260810 00:48:15.077912 15391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:48:15.081969 15391 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:48:15.083606 15391 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5: Loaded TSK: 0
I20260810 00:48:15.085168 15391 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 00:48:15.090394 15372 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260810 00:48:15.423234 15374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:48:15.423552 15374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:48:15.423710 15374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:48:15.431026 15374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:48:15.431162 15374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:48:15.431416 15374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.1
I20260810 00:48:15.446997 15374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:43281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.6.20:46067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.1:36715
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.6.1
--webserver_port=36253
--tserver_master_addrs=127.28.6.62:33563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.1
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:48:15.448726 15374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:48:15.450501 15374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:48:15.475342 15429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:15.476136 15428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:15.487373 15433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:48:15.487877 15374 server_base.cc:1034] running on GCE node
I20260810 00:48:15.488582 15374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:48:15.489727 15374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:48:15.491048 15374 hybrid_clock.cc:648] HybridClock initialized: now 1786322895490981 us; error 53 us; skew 500 ppm
I20260810 00:48:15.491504 15374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:48:15 dist-test-slave-1nl6 krb5kdc[13762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322895, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:48:15.495216 15374 init.cc:377] Logged in from keytab as kudu/127.28.6.1@KRBTEST.COM (short username kudu)
I20260810 00:48:15.500196 15374 webserver.cc:460] Webserver started at http://127.28.6.1:36253/ using document root <none> and password file <none>
I20260810 00:48:15.500990 15374 fs_manager.cc:359] Metadata directory not provided
I20260810 00:48:15.501161 15374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 00:48:15 dist-test-slave-1nl6 krb5kdc[13762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322895, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for HTTP/127.28.6.20@KRBTEST.COM
I20260810 00:48:15.539157 15374 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 00:48:15.555480 15440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:15.557377 15374 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.009s sys 0.007s
I20260810 00:48:15.557636 15374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-0/wal
uuid: "aea2826c1e1d4742801c756244bcaf2c"
format_stamp: "Formatted at 2026-08-10 00:48:12 on dist-test-slave-1nl6"
server_key: "4bffa9affe67f2ed29e4177c894b59c0"
server_key_iv: "a6a555fc80855515caf1f83c952ba490"
server_key_version: "kuduclusterkey@0"
I20260810 00:48:15.559193 15374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:48:15.585860 15374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:48:15.591239 15374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:48:15.591611 15374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:48:15.598886 15374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:48:15.624547 15374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:48:15.624725 15374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:15.624900 15374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:48:15.625029 15374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:15.681142 15374 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.1:36715
I20260810 00:48:15.681238 15622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.1:36715 every 8 connection(s)
I20260810 00:48:15.683182 15374 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
I20260810 00:48:15.689540 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 15374
I20260810 00:48:15.690438 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.6.20:46067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.2:33139
--local_ip_for_outbound_sockets=127.28.6.2
--tserver_master_addrs=127.28.6.62:33563
--webserver_port=46603
--webserver_interface=127.28.6.2
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:43281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.6.20:46067/kms with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 00:48:15 dist-test-slave-1nl6 krb5kdc[13762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322895, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:48:15.745548 15626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:15.694114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:33563 (local address 127.28.6.1:34211)
0810 00:48:15.719228 (+ 25114us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:15.719301 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:15.721847 (+ 2546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:15.723821 (+ 1974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:15.723834 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:15.724425 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:48:15.725000 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:15.725018 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:15.726742 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:15.726747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:15.728046 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:15.728059 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:15.729425 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:15.729430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:15.729610 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:15.733547 (+ 3937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:48:15.733584 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:48:15.736807 (+ 3223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:48:15.741124 (+ 4317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:15.741133 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:15.741148 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:15.741567 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:15.742019 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:15.742026 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:15.742030 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:15.742263 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:15.743082 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:48:15.743089 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:48:15.743405 (+ 316us) client_negotiation.cc:769] Sending connection context
0810 00:48:15.743674 (+ 269us) client_negotiation.cc:240] Negotiation successful
0810 00:48:15.744228 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24740,"thread_start_us":27377,"threads_started":1}
I20260810 00:48:15.745548 15629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:15.695330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:34211 (local address 127.28.6.62:33563)
0810 00:48:15.718263 (+ 22933us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:15.718290 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:15.721576 (+ 3286us) server_negotiation.cc:407] Connection header received
0810 00:48:15.722692 (+ 1116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:15.722704 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:15.723223 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:15.723653 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:15.725222 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:15.726487 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:15.728194 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:15.728867 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:15.729083 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:15.737125 (+ 8042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:15.737136 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:15.737155 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:15.737206 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:15.740974 (+ 3768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:15.741729 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:15.741742 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:15.741748 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:15.741896 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:15.742392 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:15.742396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:15.742398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:15.742884 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:15.743154 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:15.743918 (+ 764us) server_negotiation.cc:299] Negotiation successful
0810 00:48:15.744287 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22553,"thread_start_us":22892,"threads_started":1}
I20260810 00:48:15.749639 15624 heartbeater.cc:347] Connected to a master server at 127.28.6.62:33563
I20260810 00:48:15.749954 15624 heartbeater.cc:464] Registering TS with master...
I20260810 00:48:15.750872 15624 heartbeater.cc:511] Master 127.28.6.62:33563 requested a full tablet report, sending...
W20260810 00:48:15.754130 15624 heartbeater.cc:644] Failed to heartbeat to 127.28.6.62:33563 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 00:48:15.754371 15624 heartbeater.cc:464] Registering TS with master...
I20260810 00:48:15.754647 15624 heartbeater.cc:511] Master 127.28.6.62:33563 requested a full tablet report, sending...
I20260810 00:48:15.756086 15624 heartbeater.cc:464] Registering TS with master...
I20260810 00:48:15.756455 15624 heartbeater.cc:511] Master 127.28.6.62:33563 requested a full tablet report, sending...
W20260810 00:48:15.757576 15624 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 00:48:16.175809 15630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:48:16.176182 15630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:48:16.176362 15630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:48:16.187907 15630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:48:16.188040 15630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:48:16.188218 15630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.2
I20260810 00:48:16.203521 15630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:43281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.6.20:46067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.2:33139
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.6.2
--webserver_port=46603
--tserver_master_addrs=127.28.6.62:33563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.2
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:48:16.205333 15630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:48:16.207134 15630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:48:16.223001 15650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:16.223729 15652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:16.223763 15649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:48:16.224243 15630 server_base.cc:1034] running on GCE node
I20260810 00:48:16.224864 15630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:48:16.225914 15630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:48:16.227149 15630 hybrid_clock.cc:648] HybridClock initialized: now 1786322896227085 us; error 55 us; skew 500 ppm
I20260810 00:48:16.227561 15630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:48:16 dist-test-slave-1nl6 krb5kdc[13762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322896, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:48:16.230839 15630 init.cc:377] Logged in from keytab as kudu/127.28.6.2@KRBTEST.COM (short username kudu)
I20260810 00:48:16.232360 15630 webserver.cc:460] Webserver started at http://127.28.6.2:46603/ using document root <none> and password file <none>
I20260810 00:48:16.233105 15630 fs_manager.cc:359] Metadata directory not provided
I20260810 00:48:16.233269 15630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 00:48:16 dist-test-slave-1nl6 krb5kdc[13762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322896, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for HTTP/127.28.6.20@KRBTEST.COM
I20260810 00:48:16.258054 15630 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 00:48:16.262506 15669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:16.264262 15630 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 00:48:16.264501 15630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-1/wal
uuid: "1f0de896a4134165b1ab0c26f803f26b"
format_stamp: "Formatted at 2026-08-10 00:48:13 on dist-test-slave-1nl6"
server_key: "7fdcf2d61adc7166f9efb945ca8d8cb2"
server_key_iv: "23d8f33b3e2d4df233b194d87f8feeda"
server_key_version: "kuduclusterkey@0"
I20260810 00:48:16.265960 15630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:48:16.295190 15630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:48:16.300238 15630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:48:16.300587 15630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:48:16.301736 15630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:48:16.303462 15630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:48:16.303601 15630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:16.303717 15630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:48:16.303831 15630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:16.402685 15630 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.2:33139
I20260810 00:48:16.402733 15843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.2:33139 every 8 connection(s)
I20260810 00:48:16.404398 15630 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
I20260810 00:48:16.410121 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 15630
I20260810 00:48:16.410930 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.6.20:46067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.3:46159
--local_ip_for_outbound_sockets=127.28.6.3
--tserver_master_addrs=127.28.6.62:33563
--webserver_port=46839
--webserver_interface=127.28.6.3
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:43281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.6.20:46067/kms with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 00:48:16 dist-test-slave-1nl6 krb5kdc[13762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322896, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:48:16.435591 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:16.409124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:36805 (local address 127.28.6.62:33563)
0810 00:48:16.411821 (+ 2697us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:16.411827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:16.411851 (+ 24us) server_negotiation.cc:407] Connection header received
0810 00:48:16.415387 (+ 3536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:16.415393 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:16.415495 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:16.415613 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:16.417801 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:16.419034 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:16.421463 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:16.422110 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:16.422210 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:16.427996 (+ 5786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:16.428004 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:16.428009 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:16.428050 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:16.431809 (+ 3759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:16.432770 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:16.432776 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:16.432780 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:16.432947 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:16.433506 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:16.433512 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:16.433516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:16.433871 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:16.433987 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:16.434988 (+ 1001us) server_negotiation.cc:299] Negotiation successful
0810 00:48:16.435215 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2570,"thread_start_us":2453,"threads_started":1}
I20260810 00:48:16.438135 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:16.408370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:33563 (local address 127.28.6.2:36805)
0810 00:48:16.410783 (+ 2413us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:16.410838 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:16.415046 (+ 4208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:16.416125 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:16.416136 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:16.416828 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:48:16.417587 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:16.417614 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:16.419259 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:16.419263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:16.421287 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:16.421303 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:16.422597 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:16.422607 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:16.422913 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:16.423698 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:48:16.423736 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:48:16.427724 (+ 3988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:48:16.432012 (+ 4288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:16.432021 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:16.432038 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:16.432624 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:16.433106 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:16.433111 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:16.433115 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:16.433364 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:16.434096 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:48:16.434104 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:48:16.434525 (+ 421us) client_negotiation.cc:769] Sending connection context
0810 00:48:16.436179 (+ 1654us) client_negotiation.cc:240] Negotiation successful
0810 00:48:16.436755 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2128,"thread_start_us":1868,"threads_started":1}
I20260810 00:48:16.440233 15844 heartbeater.cc:347] Connected to a master server at 127.28.6.62:33563
I20260810 00:48:16.440558 15844 heartbeater.cc:464] Registering TS with master...
I20260810 00:48:16.441463 15844 heartbeater.cc:511] Master 127.28.6.62:33563 requested a full tablet report, sending...
I20260810 00:48:16.443782 15322 ts_manager.cc:194] Registered new tserver with Master: 1f0de896a4134165b1ab0c26f803f26b (127.28.6.2:33139)
I20260810 00:48:16.446228 15322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.2@KRBTEST.COM'} at 127.28.6.2:36805
Aug 10 00:48:16 dist-test-slave-1nl6 krb5kdc[13762](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786322896, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:48:16.757723 15624 heartbeater.cc:464] Registering TS with master...
I20260810 00:48:16.758199 15624 heartbeater.cc:511] Master 127.28.6.62:33563 requested a full tablet report, sending...
I20260810 00:48:16.759503 15322 ts_manager.cc:194] Registered new tserver with Master: aea2826c1e1d4742801c756244bcaf2c (127.28.6.1:36715)
I20260810 00:48:16.760789 15322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.1@KRBTEST.COM'} at 127.28.6.1:34211
Aug 10 00:48:16 dist-test-slave-1nl6 krb5kdc[13762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322896, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for HTTP/127.28.6.20@KRBTEST.COM
W20260810 00:48:16.916795 15851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:48:16.917207 15851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:48:16.917389 15851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:48:16.927438 15851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:48:16.927585 15851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:48:16.927790 15851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.3
I20260810 00:48:16.937887 15851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:43281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.6.20:46067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.3:46159
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.6.3
--webserver_port=46839
--tserver_master_addrs=127.28.6.62:33563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.3
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:48:16.939352 15851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:48:16.940640 15851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:48:16.954867 15863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:16.956135 15862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:16.957023 15865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:48:16.957489 15851 server_base.cc:1034] running on GCE node
I20260810 00:48:16.958235 15851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:48:16.959482 15851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:48:16.960817 15851 hybrid_clock.cc:648] HybridClock initialized: now 1786322896960752 us; error 45 us; skew 500 ppm
I20260810 00:48:16.961237 15851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:48:16 dist-test-slave-1nl6 krb5kdc[13762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322896, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:48:16.965682 15851 init.cc:377] Logged in from keytab as kudu/127.28.6.3@KRBTEST.COM (short username kudu)
I20260810 00:48:16.971055 15851 webserver.cc:460] Webserver started at http://127.28.6.3:46839/ using document root <none> and password file <none>
I20260810 00:48:16.971987 15851 fs_manager.cc:359] Metadata directory not provided
I20260810 00:48:16.972139 15851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 00:48:16 dist-test-slave-1nl6 krb5kdc[13762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322896, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for HTTP/127.28.6.20@KRBTEST.COM
I20260810 00:48:17.024905 15851 fs_manager.cc:709] Time spent opening directory manager: real 0.032s user 0.012s sys 0.016s
I20260810 00:48:17.039529 15875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:17.041689 15851 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.001s sys 0.011s
I20260810 00:48:17.041977 15851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-2/wal
uuid: "30d3f8306f584df2b93842c33e43bfb0"
format_stamp: "Formatted at 2026-08-10 00:48:13 on dist-test-slave-1nl6"
server_key: "0e55d7b33ad034b6186bfd5a97d6dfde"
server_key_iv: "6ad8df0ccc3452990f68203d29eae85c"
server_key_version: "kuduclusterkey@0"
I20260810 00:48:17.043622 15851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:48:17.101114 15851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:48:17.109341 15851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:48:17.109752 15851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:48:17.120785 15851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:48:17.123632 15851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:48:17.123761 15851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:17.123878 15851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:48:17.123992 15851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Aug 10 00:48:17 dist-test-slave-1nl6 krb5kdc[13762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322896, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for HTTP/127.28.6.20@KRBTEST.COM
I20260810 00:48:17.313444 15851 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.3:46159
I20260810 00:48:17.313482 16058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.3:46159 every 8 connection(s)
I20260810 00:48:17.315572 15851 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
I20260810 00:48:17.327303 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 15851
Aug 10 00:48:17 dist-test-slave-1nl6 krb5kdc[13762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322896, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:48:17.361040 16065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:17.328686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49606 (local address 127.28.6.62:33563)
0810 00:48:17.340251 (+ 11565us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:17.340258 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:17.341059 (+ 801us) server_negotiation.cc:407] Connection header received
0810 00:48:17.341121 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:17.341127 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:17.341208 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:17.341307 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:17.343695 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:17.345029 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:17.347089 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:17.347819 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:17.347928 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:17.353010 (+ 5082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:17.353019 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:17.353024 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:17.353084 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:17.357107 (+ 4023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:17.358112 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:17.358118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:17.358122 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:17.358366 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:17.358816 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:17.358821 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:17.358825 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:17.359232 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:17.359501 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:17.360399 (+ 898us) server_negotiation.cc:299] Negotiation successful
0810 00:48:17.360638 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11384,"thread_start_us":11243,"threads_started":1}
I20260810 00:48:17.367938 16062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:17.319565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:37299 (local address 127.28.6.62:33563)
0810 00:48:17.320115 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:17.320122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:17.321203 (+ 1081us) server_negotiation.cc:407] Connection header received
0810 00:48:17.323857 (+ 2654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:17.323865 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:17.323952 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:17.324060 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:17.326241 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:17.327444 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:17.340117 (+ 12673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:17.340823 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:17.342166 (+ 1343us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:17.349778 (+ 7612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:17.349791 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:17.349800 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:17.349848 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:17.356623 (+ 6775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:17.364660 (+ 8037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:17.364664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:17.364667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:17.364809 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:17.365392 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:17.365397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:17.365400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:17.365774 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:17.367136 (+ 1362us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:17.367295 (+ 159us) server_negotiation.cc:299] Negotiation successful
0810 00:48:17.367528 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":262,"threads_started":1}
I20260810 00:48:17.369138 16063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:17.320148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:33563 (local address 127.28.6.3:37299)
0810 00:48:17.320997 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:17.321055 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:17.323427 (+ 2372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:17.324549 (+ 1122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:17.324560 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:17.325211 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:48:17.325998 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:17.326026 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:17.327656 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:17.327662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:17.339553 (+ 11891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:17.339580 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:17.341382 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:17.341392 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:17.341706 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:17.342570 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:48:17.342602 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:48:17.349412 (+ 6810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:48:17.363703 (+ 14291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:17.363715 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:17.363737 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:17.364470 (+ 733us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:17.364964 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:17.364970 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:17.364973 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:17.365244 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:17.365957 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:48:17.365966 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:48:17.366419 (+ 453us) client_negotiation.cc:769] Sending connection context
0810 00:48:17.366734 (+ 315us) client_negotiation.cc:240] Negotiation successful
0810 00:48:17.367370 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":325,"threads_started":1}
I20260810 00:48:17.370985 16060 heartbeater.cc:347] Connected to a master server at 127.28.6.62:33563
I20260810 00:48:17.371308 16060 heartbeater.cc:464] Registering TS with master...
I20260810 00:48:17.372283 16060 heartbeater.cc:511] Master 127.28.6.62:33563 requested a full tablet report, sending...
I20260810 00:48:17.373962 15322 ts_manager.cc:194] Registered new tserver with Master: 30d3f8306f584df2b93842c33e43bfb0 (127.28.6.3:46159)
I20260810 00:48:17.375202 15322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.3@KRBTEST.COM'} at 127.28.6.3:37299
I20260810 00:48:17.378855 28696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 00:48:17.409005 16062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:17.383155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56114 (local address 127.28.6.62:33563)
0810 00:48:17.383322 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:17.383328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:17.391481 (+ 8153us) server_negotiation.cc:407] Connection header received
0810 00:48:17.391664 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:17.391670 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:17.391769 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:17.391886 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:17.393922 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:17.395057 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:17.397061 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:17.397745 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:17.397865 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:17.401368 (+ 3503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:17.401375 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:17.401380 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:17.401423 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:17.405046 (+ 3623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:17.405901 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:17.405907 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:17.405910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:17.406066 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:17.406565 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:17.406570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:17.406574 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:17.406965 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:17.407237 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:17.408423 (+ 1186us) server_negotiation.cc:299] Negotiation successful
0810 00:48:17.408649 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 00:48:17.415489 15322 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56114:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 00:48:17.427428 15322 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 00:48:17.451690 15844 heartbeater.cc:503] Master 127.28.6.62:33563 was elected leader, sending a full tablet report...
I20260810 00:48:17.489007 16091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:17.452587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:46159 (local address 127.0.0.1:32998)
0810 00:48:17.477001 (+ 24414us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:17.477058 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:17.477306 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:17.478655 (+ 1349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:17.478666 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:17.478723 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:48:17.479270 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:17.479287 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:17.481333 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:17.481341 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:17.484670 (+ 3329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:17.484694 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:17.487727 (+ 3033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:17.487737 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:17.487966 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:17.488017 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 00:48:17.488148 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 00:48:17.488444 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24058,"mutex_wait_us":122,"thread_start_us":28750,"threads_started":1}
I20260810 00:48:17.490329 16090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:17.452805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:33139 (local address 127.0.0.1:52332)
0810 00:48:17.481341 (+ 28536us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:17.481364 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:17.481517 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:17.482506 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:17.482511 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:17.482537 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:48:17.483316 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:17.483332 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:17.485245 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:17.485250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:17.487297 (+ 2047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:17.487314 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:17.489489 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:17.489496 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:17.489668 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:17.489707 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 00:48:17.489797 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 00:48:17.489929 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28408,"mutex_wait_us":1,"thread_start_us":23680,"threads_started":1}
I20260810 00:48:17.490875 16096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:17.453438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32998 (local address 127.28.6.3:46159)
0810 00:48:17.476022 (+ 22584us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:17.476037 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:17.477276 (+ 1239us) server_negotiation.cc:407] Connection header received
0810 00:48:17.477646 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:17.477659 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:17.477997 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:17.478214 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:48:17.479507 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:17.481172 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:17.484914 (+ 3742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:17.486998 (+ 2084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:17.487164 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:17.487533 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:17.489937 (+ 2404us) server_negotiation.cc:299] Negotiation successful
0810 00:48:17.490176 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22313,"thread_start_us":22611,"threads_started":1}
I20260810 00:48:17.491382 16094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:17.453205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52332 (local address 127.28.6.2:33139)
0810 00:48:17.480920 (+ 27715us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:17.480928 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:17.481661 (+ 733us) server_negotiation.cc:407] Connection header received
0810 00:48:17.481762 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:17.481770 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:17.482002 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:17.482253 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:48:17.483523 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:17.485046 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:17.487495 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:17.488953 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:17.489083 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:17.489323 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:17.490859 (+ 1536us) server_negotiation.cc:299] Negotiation successful
0810 00:48:17.491046 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27541,"thread_start_us":27359,"threads_started":1}
I20260810 00:48:17.495925 15746 tablet_service.cc:1467] Processing CreateTablet for tablet 5dc3894876834b3485668df673cb6097 (DEFAULT_TABLE table=test-table [id=c41b2c6152094656afa28fae98956114]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:48:17.497610 15746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dc3894876834b3485668df673cb6097. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:48:17.497593 15948 tablet_service.cc:1467] Processing CreateTablet for tablet 5dc3894876834b3485668df673cb6097 (DEFAULT_TABLE table=test-table [id=c41b2c6152094656afa28fae98956114]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:48:17.499511 15948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dc3894876834b3485668df673cb6097. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:48:17.501060 16091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:17.487975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:36715 (local address 127.0.0.1:53900)
0810 00:48:17.489604 (+ 1629us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:17.489628 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:17.489739 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:17.494189 (+ 4450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:17.494195 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:17.494223 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:48:17.494384 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:17.494396 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:17.496394 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:17.496400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:17.498217 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:17.498233 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:17.500333 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:17.500340 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:17.500516 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:17.500549 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 00:48:17.500644 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 00:48:17.500786 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1499,"thread_start_us":4745,"threads_started":1}
I20260810 00:48:17.501838 16097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:17.482028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53900 (local address 127.28.6.1:36715)
0810 00:48:17.493365 (+ 11337us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:17.493373 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:17.493402 (+ 29us) server_negotiation.cc:407] Connection header received
0810 00:48:17.493495 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:17.493503 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:17.493744 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:17.493937 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:48:17.494589 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:17.496203 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:17.498405 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:17.499873 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:17.499997 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:17.500231 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:17.501309 (+ 1078us) server_negotiation.cc:299] Negotiation successful
0810 00:48:17.501496 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11161,"thread_start_us":11081,"threads_started":1}
I20260810 00:48:17.505878 15520 tablet_service.cc:1467] Processing CreateTablet for tablet 5dc3894876834b3485668df673cb6097 (DEFAULT_TABLE table=test-table [id=c41b2c6152094656afa28fae98956114]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:48:17.507277 15520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dc3894876834b3485668df673cb6097. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:48:17.521925 16102 tablet_bootstrap.cc:492] T 5dc3894876834b3485668df673cb6097 P 30d3f8306f584df2b93842c33e43bfb0: Bootstrap starting.
I20260810 00:48:17.527704 16100 tablet_bootstrap.cc:492] T 5dc3894876834b3485668df673cb6097 P 1f0de896a4134165b1ab0c26f803f26b: Bootstrap starting.
I20260810 00:48:17.529191 16103 tablet_bootstrap.cc:492] T 5dc3894876834b3485668df673cb6097 P aea2826c1e1d4742801c756244bcaf2c: Bootstrap starting.
I20260810 00:48:17.530867 16102 tablet_bootstrap.cc:654] T 5dc3894876834b3485668df673cb6097 P 30d3f8306f584df2b93842c33e43bfb0: Neither blocks nor log segments found. Creating new log.
I20260810 00:48:17.531813 16100 tablet_bootstrap.cc:654] T 5dc3894876834b3485668df673cb6097 P 1f0de896a4134165b1ab0c26f803f26b: Neither blocks nor log segments found. Creating new log.
I20260810 00:48:17.532353 16102 log.cc:824] T 5dc3894876834b3485668df673cb6097 P 30d3f8306f584df2b93842c33e43bfb0: Log is configured to *not* fsync() on all Append() calls
I20260810 00:48:17.533013 16100 log.cc:824] T 5dc3894876834b3485668df673cb6097 P 1f0de896a4134165b1ab0c26f803f26b: Log is configured to *not* fsync() on all Append() calls
I20260810 00:48:17.533984 16103 tablet_bootstrap.cc:654] T 5dc3894876834b3485668df673cb6097 P aea2826c1e1d4742801c756244bcaf2c: Neither blocks nor log segments found. Creating new log.
I20260810 00:48:17.535501 16103 log.cc:824] T 5dc3894876834b3485668df673cb6097 P aea2826c1e1d4742801c756244bcaf2c: Log is configured to *not* fsync() on all Append() calls
I20260810 00:48:17.544112 16100 tablet_bootstrap.cc:492] T 5dc3894876834b3485668df673cb6097 P 1f0de896a4134165b1ab0c26f803f26b: No bootstrap required, opened a new log
I20260810 00:48:17.544457 16100 ts_tablet_manager.cc:1397] T 5dc3894876834b3485668df673cb6097 P 1f0de896a4134165b1ab0c26f803f26b: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.008s
I20260810 00:48:17.544703 16103 tablet_bootstrap.cc:492] T 5dc3894876834b3485668df673cb6097 P aea2826c1e1d4742801c756244bcaf2c: No bootstrap required, opened a new log
I20260810 00:48:17.545053 16103 ts_tablet_manager.cc:1397] T 5dc3894876834b3485668df673cb6097 P aea2826c1e1d4742801c756244bcaf2c: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.005s
I20260810 00:48:17.545346 16102 tablet_bootstrap.cc:492] T 5dc3894876834b3485668df673cb6097 P 30d3f8306f584df2b93842c33e43bfb0: No bootstrap required, opened a new log
I20260810 00:48:17.545651 16102 ts_tablet_manager.cc:1397] T 5dc3894876834b3485668df673cb6097 P 30d3f8306f584df2b93842c33e43bfb0: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.006s
I20260810 00:48:17.555428 16100 raft_consensus.cc:348] T 5dc3894876834b3485668df673cb6097 P 1f0de896a4134165b1ab0c26f803f26b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d3f8306f584df2b93842c33e43bfb0" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46159 } } peers { permanent_uuid: "aea2826c1e1d4742801c756244bcaf2c" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 36715 } } peers { permanent_uuid: "1f0de896a4134165b1ab0c26f803f26b" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 33139 } }
I20260810 00:48:17.555912 16100 raft_consensus.cc:374] T 5dc3894876834b3485668df673cb6097 P 1f0de896a4134165b1ab0c26f803f26b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:48:17.556041 16100 raft_consensus.cc:729] T 5dc3894876834b3485668df673cb6097 P 1f0de896a4134165b1ab0c26f803f26b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0de896a4134165b1ab0c26f803f26b, State: Initialized, Role: FOLLOWER
I20260810 00:48:17.556650 16103 raft_consensus.cc:348] T 5dc3894876834b3485668df673cb6097 P aea2826c1e1d4742801c756244bcaf2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d3f8306f584df2b93842c33e43bfb0" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46159 } } peers { permanent_uuid: "aea2826c1e1d4742801c756244bcaf2c" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 36715 } } peers { permanent_uuid: "1f0de896a4134165b1ab0c26f803f26b" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 33139 } }
I20260810 00:48:17.556818 16100 consensus_queue.cc:260] T 5dc3894876834b3485668df673cb6097 P 1f0de896a4134165b1ab0c26f803f26b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d3f8306f584df2b93842c33e43bfb0" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46159 } } peers { permanent_uuid: "aea2826c1e1d4742801c756244bcaf2c" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 36715 } } peers { permanent_uuid: "1f0de896a4134165b1ab0c26f803f26b" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 33139 } }
I20260810 00:48:17.557173 16103 raft_consensus.cc:374] T 5dc3894876834b3485668df673cb6097 P aea2826c1e1d4742801c756244bcaf2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:48:17.557350 16103 raft_consensus.cc:729] T 5dc3894876834b3485668df673cb6097 P aea2826c1e1d4742801c756244bcaf2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aea2826c1e1d4742801c756244bcaf2c, State: Initialized, Role: FOLLOWER
I20260810 00:48:17.556502 16102 raft_consensus.cc:348] T 5dc3894876834b3485668df673cb6097 P 30d3f8306f584df2b93842c33e43bfb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d3f8306f584df2b93842c33e43bfb0" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46159 } } peers { permanent_uuid: "aea2826c1e1d4742801c756244bcaf2c" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 36715 } } peers { permanent_uuid: "1f0de896a4134165b1ab0c26f803f26b" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 33139 } }
I20260810 00:48:17.558239 16102 raft_consensus.cc:374] T 5dc3894876834b3485668df673cb6097 P 30d3f8306f584df2b93842c33e43bfb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:48:17.558369 16102 raft_consensus.cc:729] T 5dc3894876834b3485668df673cb6097 P 30d3f8306f584df2b93842c33e43bfb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30d3f8306f584df2b93842c33e43bfb0, State: Initialized, Role: FOLLOWER
I20260810 00:48:17.558182 16103 consensus_queue.cc:260] T 5dc3894876834b3485668df673cb6097 P aea2826c1e1d4742801c756244bcaf2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d3f8306f584df2b93842c33e43bfb0" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46159 } } peers { permanent_uuid: "aea2826c1e1d4742801c756244bcaf2c" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 36715 } } peers { permanent_uuid: "1f0de896a4134165b1ab0c26f803f26b" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 33139 } }
I20260810 00:48:17.559149 16102 consensus_queue.cc:260] T 5dc3894876834b3485668df673cb6097 P 30d3f8306f584df2b93842c33e43bfb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d3f8306f584df2b93842c33e43bfb0" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46159 } } peers { permanent_uuid: "aea2826c1e1d4742801c756244bcaf2c" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 36715 } } peers { permanent_uuid: "1f0de896a4134165b1ab0c26f803f26b" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 33139 } }
I20260810 00:48:17.564126 16060 heartbeater.cc:503] Master 127.28.6.62:33563 was elected leader, sending a full tablet report...
I20260810 00:48:17.564772 16102 ts_tablet_manager.cc:1428] T 5dc3894876834b3485668df673cb6097 P 30d3f8306f584df2b93842c33e43bfb0: Time spent starting tablet: real 0.019s user 0.013s sys 0.003s
I20260810 00:48:17.566082 16100 ts_tablet_manager.cc:1428] T 5dc3894876834b3485668df673cb6097 P 1f0de896a4134165b1ab0c26f803f26b: Time spent starting tablet: real 0.021s user 0.016s sys 0.004s
I20260810 00:48:17.568199 15624 heartbeater.cc:503] Master 127.28.6.62:33563 was elected leader, sending a full tablet report...
I20260810 00:48:17.568696 16103 ts_tablet_manager.cc:1428] T 5dc3894876834b3485668df673cb6097 P aea2826c1e1d4742801c756244bcaf2c: Time spent starting tablet: real 0.023s user 0.007s sys 0.011s
W20260810 00:48:17.569510 16061 tablet.cc:2369] T 5dc3894876834b3485668df673cb6097 P 30d3f8306f584df2b93842c33e43bfb0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 00:48:17.667147 15845 tablet.cc:2369] T 5dc3894876834b3485668df673cb6097 P 1f0de896a4134165b1ab0c26f803f26b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 00:48:17.693598 15625 tablet.cc:2369] T 5dc3894876834b3485668df673cb6097 P aea2826c1e1d4742801c756244bcaf2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:48:17.695541 16110 raft_consensus.cc:482] T 5dc3894876834b3485668df673cb6097 P 30d3f8306f584df2b93842c33e43bfb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:48:17.695860 16110 raft_consensus.cc:504] T 5dc3894876834b3485668df673cb6097 P 30d3f8306f584df2b93842c33e43bfb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d3f8306f584df2b93842c33e43bfb0" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46159 } } peers { permanent_uuid: "aea2826c1e1d4742801c756244bcaf2c" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 36715 } } peers { permanent_uuid: "1f0de896a4134165b1ab0c26f803f26b" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 33139 } }
I20260810 00:48:17.698037 16110 leader_election.cc:288] T 5dc3894876834b3485668df673cb6097 P 30d3f8306f584df2b93842c33e43bfb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aea2826c1e1d4742801c756244bcaf2c (127.28.6.1:36715), 1f0de896a4134165b1ab0c26f803f26b (127.28.6.2:33139)
I20260810 00:48:17.704988 16063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:17.698303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:36715 (local address 127.28.6.3:57865)
0810 00:48:17.698820 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:17.698846 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:17.699030 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:17.699729 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:17.699734 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:17.699762 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:48:17.699970 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:17.699985 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:17.701423 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:17.701429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:17.703071 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:17.703085 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:17.704239 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:17.704245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:17.704427 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:17.704463 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 00:48:17.704559 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 00:48:17.704691 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336}
I20260810 00:48:17.704999 16097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:17.699109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:57865 (local address 127.28.6.1:36715)
0810 00:48:17.699378 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:17.699384 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:17.699404 (+ 20us) server_negotiation.cc:407] Connection header received
0810 00:48:17.699479 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:17.699485 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:17.699582 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:17.699699 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:48:17.700135 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:17.701261 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:17.703237 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:17.704067 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:17.704211 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:17.704320 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:17.704601 (+ 281us) server_negotiation.cc:299] Negotiation successful
0810 00:48:17.704747 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 00:48:17.706284 16113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:17.698405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:33139 (local address 127.28.6.3:53653)
0810 00:48:17.699670 (+ 1265us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:17.699688 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:17.699893 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:17.700281 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:17.700286 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:17.700310 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:48:17.700608 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:17.700616 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:17.701753 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:17.701759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:17.703951 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:17.703967 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:17.705522 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:17.705528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:17.705682 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:17.705705 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 00:48:17.705817 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 00:48:17.705938 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1145,"thread_start_us":940,"threads_started":1}
I20260810 00:48:17.706516 15550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dc3894876834b3485668df673cb6097" candidate_uuid: "30d3f8306f584df2b93842c33e43bfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aea2826c1e1d4742801c756244bcaf2c" is_pre_election: true
I20260810 00:48:17.707062 16094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:17.698616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:53653 (local address 127.28.6.2:33139)
0810 00:48:17.698812 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:17.698819 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:17.699781 (+ 962us) server_negotiation.cc:407] Connection header received
0810 00:48:17.700072 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:17.700078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:17.700160 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:17.700261 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:48:17.700768 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:17.701588 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:17.704140 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:17.705302 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:17.705452 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:17.705561 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:17.705848 (+ 287us) server_negotiation.cc:299] Negotiation successful
0810 00:48:17.706802 (+ 954us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 00:48:17.707170 15550 raft_consensus.cc:2454] T 5dc3894876834b3485668df673cb6097 P aea2826c1e1d4742801c756244bcaf2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30d3f8306f584df2b93842c33e43bfb0 in term 0.
I20260810 00:48:17.707980 15877 leader_election.cc:302] T 5dc3894876834b3485668df673cb6097 P 30d3f8306f584df2b93842c33e43bfb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30d3f8306f584df2b93842c33e43bfb0, aea2826c1e1d4742801c756244bcaf2c; no voters:
I20260810 00:48:17.708083 15774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dc3894876834b3485668df673cb6097" candidate_uuid: "30d3f8306f584df2b93842c33e43bfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0de896a4134165b1ab0c26f803f26b" is_pre_election: true
I20260810 00:48:17.708432 16110 raft_consensus.cc:2781] T 5dc3894876834b3485668df673cb6097 P 30d3f8306f584df2b93842c33e43bfb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 00:48:17.708595 16110 raft_consensus.cc:482] T 5dc3894876834b3485668df673cb6097 P 30d3f8306f584df2b93842c33e43bfb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 00:48:17.708741 16110 raft_consensus.cc:3037] T 5dc3894876834b3485668df673cb6097 P 30d3f8306f584df2b93842c33e43bfb0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:48:17.708667 15774 raft_consensus.cc:2454] T 5dc3894876834b3485668df673cb6097 P 1f0de896a4134165b1ab0c26f803f26b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30d3f8306f584df2b93842c33e43bfb0 in term 0.
I20260810 00:48:17.711356 16110 raft_consensus.cc:504] T 5dc3894876834b3485668df673cb6097 P 30d3f8306f584df2b93842c33e43bfb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d3f8306f584df2b93842c33e43bfb0" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46159 } } peers { permanent_uuid: "aea2826c1e1d4742801c756244bcaf2c" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 36715 } } peers { permanent_uuid: "1f0de896a4134165b1ab0c26f803f26b" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 33139 } }
I20260810 00:48:17.712000 16110 leader_election.cc:288] T 5dc3894876834b3485668df673cb6097 P 30d3f8306f584df2b93842c33e43bfb0 [CANDIDATE]: Term 1 election: Requested vote from peers aea2826c1e1d4742801c756244bcaf2c (127.28.6.1:36715), 1f0de896a4134165b1ab0c26f803f26b (127.28.6.2:33139)
I20260810 00:48:17.712488 15550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dc3894876834b3485668df673cb6097" candidate_uuid: "30d3f8306f584df2b93842c33e43bfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aea2826c1e1d4742801c756244bcaf2c"
I20260810 00:48:17.712567 15774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dc3894876834b3485668df673cb6097" candidate_uuid: "30d3f8306f584df2b93842c33e43bfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0de896a4134165b1ab0c26f803f26b"
I20260810 00:48:17.712706 15550 raft_consensus.cc:3037] T 5dc3894876834b3485668df673cb6097 P aea2826c1e1d4742801c756244bcaf2c [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:48:17.712786 15774 raft_consensus.cc:3037] T 5dc3894876834b3485668df673cb6097 P 1f0de896a4134165b1ab0c26f803f26b [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:48:17.714987 15550 raft_consensus.cc:2454] T 5dc3894876834b3485668df673cb6097 P aea2826c1e1d4742801c756244bcaf2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30d3f8306f584df2b93842c33e43bfb0 in term 1.
I20260810 00:48:17.715390 15877 leader_election.cc:302] T 5dc3894876834b3485668df673cb6097 P 30d3f8306f584df2b93842c33e43bfb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30d3f8306f584df2b93842c33e43bfb0, aea2826c1e1d4742801c756244bcaf2c; no voters:
I20260810 00:48:17.715699 16110 raft_consensus.cc:2781] T 5dc3894876834b3485668df673cb6097 P 30d3f8306f584df2b93842c33e43bfb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:48:17.716153 15774 raft_consensus.cc:2454] T 5dc3894876834b3485668df673cb6097 P 1f0de896a4134165b1ab0c26f803f26b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30d3f8306f584df2b93842c33e43bfb0 in term 1.
I20260810 00:48:17.716270 16110 raft_consensus.cc:686] T 5dc3894876834b3485668df673cb6097 P 30d3f8306f584df2b93842c33e43bfb0 [term 1 LEADER]: Becoming Leader. State: Replica: 30d3f8306f584df2b93842c33e43bfb0, State: Running, Role: LEADER
I20260810 00:48:17.717041 16110 consensus_queue.cc:237] T 5dc3894876834b3485668df673cb6097 P 30d3f8306f584df2b93842c33e43bfb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d3f8306f584df2b93842c33e43bfb0" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46159 } } peers { permanent_uuid: "aea2826c1e1d4742801c756244bcaf2c" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 36715 } } peers { permanent_uuid: "1f0de896a4134165b1ab0c26f803f26b" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 33139 } }
I20260810 00:48:17.722195 15321 catalog_manager.cc:5495] T 5dc3894876834b3485668df673cb6097 P 30d3f8306f584df2b93842c33e43bfb0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30d3f8306f584df2b93842c33e43bfb0 (127.28.6.3). New cstate: current_term: 1 leader_uuid: "30d3f8306f584df2b93842c33e43bfb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d3f8306f584df2b93842c33e43bfb0" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aea2826c1e1d4742801c756244bcaf2c" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 36715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0de896a4134165b1ab0c26f803f26b" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 33139 } health_report { overall_health: UNKNOWN } } }
I20260810 00:48:17.743745 16096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:17.737950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33004 (local address 127.28.6.3:46159)
0810 00:48:17.738162 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:17.738169 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:17.738305 (+ 136us) server_negotiation.cc:407] Connection header received
0810 00:48:17.738559 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:17.738563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:17.738646 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:17.738811 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:48:17.739226 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:17.739981 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:17.741511 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:17.741912 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:17.742047 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:17.742565 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:48:17.743129 (+ 564us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:48:17.743285 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:17.743399 (+ 114us) server_negotiation.cc:299] Negotiation successful
0810 00:48:17.743479 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 00:48:17.752261 15550 raft_consensus.cc:1261] T 5dc3894876834b3485668df673cb6097 P aea2826c1e1d4742801c756244bcaf2c [term 1 FOLLOWER]: Refusing update from remote peer 30d3f8306f584df2b93842c33e43bfb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:48:17.752262 15774 raft_consensus.cc:1261] T 5dc3894876834b3485668df673cb6097 P 1f0de896a4134165b1ab0c26f803f26b [term 1 FOLLOWER]: Refusing update from remote peer 30d3f8306f584df2b93842c33e43bfb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:48:17.753294 16116 consensus_queue.cc:1035] T 5dc3894876834b3485668df673cb6097 P 30d3f8306f584df2b93842c33e43bfb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aea2826c1e1d4742801c756244bcaf2c" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 36715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:48:17.753666 16110 consensus_queue.cc:1035] T 5dc3894876834b3485668df673cb6097 P 30d3f8306f584df2b93842c33e43bfb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0de896a4134165b1ab0c26f803f26b" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 33139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:48:17.780270 15322 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:48:17.780565 15322 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:48:17.784045 15322 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72da27ef4e3d40da825620568c4face5: Sending DeleteTablet for 3 replicas of tablet 5dc3894876834b3485668df673cb6097
I20260810 00:48:17.785274 15746 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dc3894876834b3485668df673cb6097 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:48:17 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:52332
I20260810 00:48:17.785344 15948 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dc3894876834b3485668df673cb6097 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:48:17 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:32998
I20260810 00:48:17.786171 16137 tablet_replica.cc:330] T 5dc3894876834b3485668df673cb6097 P 1f0de896a4134165b1ab0c26f803f26b: stopping tablet replica
I20260810 00:48:17.786266 15520 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dc3894876834b3485668df673cb6097 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:48:17 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:53900
I20260810 00:48:17.786619 16137 raft_consensus.cc:2229] T 5dc3894876834b3485668df673cb6097 P 1f0de896a4134165b1ab0c26f803f26b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:48:17.787094 16137 raft_consensus.cc:2258] T 5dc3894876834b3485668df673cb6097 P 1f0de896a4134165b1ab0c26f803f26b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:48:17.787904 16136 tablet_replica.cc:330] T 5dc3894876834b3485668df673cb6097 P 30d3f8306f584df2b93842c33e43bfb0: stopping tablet replica
I20260810 00:48:17.788314 16136 raft_consensus.cc:2229] T 5dc3894876834b3485668df673cb6097 P 30d3f8306f584df2b93842c33e43bfb0 [term 1 LEADER]: Raft consensus shutting down.
I20260810 00:48:17.788955 16136 raft_consensus.cc:2258] T 5dc3894876834b3485668df673cb6097 P 30d3f8306f584df2b93842c33e43bfb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:48:17.789908 16137 ts_tablet_manager.cc:1905] T 5dc3894876834b3485668df673cb6097 P 1f0de896a4134165b1ab0c26f803f26b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:48:17.790745 16136 ts_tablet_manager.cc:1905] T 5dc3894876834b3485668df673cb6097 P 30d3f8306f584df2b93842c33e43bfb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:48:17.797392 16137 ts_tablet_manager.cc:1918] T 5dc3894876834b3485668df673cb6097 P 1f0de896a4134165b1ab0c26f803f26b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:48:17.797653 16137 log.cc:1196] T 5dc3894876834b3485668df673cb6097 P 1f0de896a4134165b1ab0c26f803f26b: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-1/wal/wals/5dc3894876834b3485668df673cb6097
I20260810 00:48:17.798056 16136 ts_tablet_manager.cc:1918] T 5dc3894876834b3485668df673cb6097 P 30d3f8306f584df2b93842c33e43bfb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:48:17.798305 16136 log.cc:1196] T 5dc3894876834b3485668df673cb6097 P 30d3f8306f584df2b93842c33e43bfb0: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-2/wal/wals/5dc3894876834b3485668df673cb6097
I20260810 00:48:17.798331 16137 ts_tablet_manager.cc:1939] T 5dc3894876834b3485668df673cb6097 P 1f0de896a4134165b1ab0c26f803f26b: Deleting consensus metadata
I20260810 00:48:17.798710 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 15374
I20260810 00:48:17.798988 16136 ts_tablet_manager.cc:1939] T 5dc3894876834b3485668df673cb6097 P 30d3f8306f584df2b93842c33e43bfb0: Deleting consensus metadata
I20260810 00:48:17.800940 15299 catalog_manager.cc:4841] TS 30d3f8306f584df2b93842c33e43bfb0 (127.28.6.3:46159): tablet 5dc3894876834b3485668df673cb6097 (table test-table [id=c41b2c6152094656afa28fae98956114]) successfully deleted
I20260810 00:48:17.801396 15300 catalog_manager.cc:4841] TS 1f0de896a4134165b1ab0c26f803f26b (127.28.6.2:33139): tablet 5dc3894876834b3485668df673cb6097 (table test-table [id=c41b2c6152094656afa28fae98956114]) successfully deleted
I20260810 00:48:17.807088 16139 tablet_replica.cc:330] T 5dc3894876834b3485668df673cb6097 P aea2826c1e1d4742801c756244bcaf2c: stopping tablet replica
I20260810 00:48:17.807523 16139 raft_consensus.cc:2229] T 5dc3894876834b3485668df673cb6097 P aea2826c1e1d4742801c756244bcaf2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:48:17.808058 16139 raft_consensus.cc:2258] T 5dc3894876834b3485668df673cb6097 P aea2826c1e1d4742801c756244bcaf2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:48:17.809924 16139 ts_tablet_manager.cc:1905] T 5dc3894876834b3485668df673cb6097 P aea2826c1e1d4742801c756244bcaf2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:48:17.815575 16139 ts_tablet_manager.cc:1918] T 5dc3894876834b3485668df673cb6097 P aea2826c1e1d4742801c756244bcaf2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:48:17.815760 16139 log.cc:1196] T 5dc3894876834b3485668df673cb6097 P aea2826c1e1d4742801c756244bcaf2c: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786322687226232-28696-0/minicluster-data/ts-0/wal/wals/5dc3894876834b3485668df673cb6097
I20260810 00:48:17.816336 16139 ts_tablet_manager.cc:1939] T 5dc3894876834b3485668df673cb6097 P aea2826c1e1d4742801c756244bcaf2c: Deleting consensus metadata
I20260810 00:48:17.817961 15299 catalog_manager.cc:4841] TS aea2826c1e1d4742801c756244bcaf2c (127.28.6.1:36715): tablet 5dc3894876834b3485668df673cb6097 (table test-table [id=c41b2c6152094656afa28fae98956114]) successfully deleted
I20260810 00:48:17.819316 16097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:17.801311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53912 (local address 127.28.6.1:36715)
0810 00:48:17.801510 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:17.801516 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:17.801547 (+ 31us) server_negotiation.cc:407] Connection header received
0810 00:48:17.801608 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:17.801615 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:17.801722 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:17.801842 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:17.803605 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:17.805137 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:17.807142 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:17.807952 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:17.808100 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:17.811580 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:17.811598 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:17.811619 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:17.811656 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:17.816058 (+ 4402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:17.816940 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:17.816949 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:17.816953 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:17.817120 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:17.817593 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:17.817598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:17.817602 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:17.818043 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:17.818353 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:17.818644 (+ 291us) server_negotiation.cc:299] Negotiation successful
0810 00:48:17.818893 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 00:48:17.819996 15618 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:48:17.945119 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 15374
I20260810 00:48:17.972543 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 15630
I20260810 00:48:17.987993 16094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:17.973433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52348 (local address 127.28.6.2:33139)
0810 00:48:17.973599 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:17.973604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:17.973623 (+ 19us) server_negotiation.cc:407] Connection header received
0810 00:48:17.973823 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:17.973827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:17.973909 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:17.973993 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:17.975829 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:17.976600 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:17.978429 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:17.978817 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:17.979144 (+ 327us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:17.982218 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:17.982238 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:17.982254 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:17.982285 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:17.985216 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:17.985993 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:17.986000 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:17.986002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:17.986140 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:17.986521 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:17.986524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:17.986526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:17.986859 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:17.987171 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:17.987496 (+ 325us) server_negotiation.cc:299] Negotiation successful
0810 00:48:17.987647 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 00:48:17.988808 15835 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:48:18.083396 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 15630
I20260810 00:48:18.113902 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 15851
I20260810 00:48:18.128561 16096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:18.114763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33018 (local address 127.28.6.3:46159)
0810 00:48:18.114995 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:18.115003 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:18.115028 (+ 25us) server_negotiation.cc:407] Connection header received
0810 00:48:18.115259 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:18.115265 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:18.115340 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:18.115464 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:18.116825 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:18.117666 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:18.119103 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:18.119572 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:18.119717 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:18.122362 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:18.122373 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:18.122390 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:18.122427 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:18.125560 (+ 3133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:18.126329 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:18.126345 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:18.126348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:18.126492 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:18.126884 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:18.126889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:18.126895 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:18.127304 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:18.127645 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:18.127954 (+ 309us) server_negotiation.cc:299] Negotiation successful
0810 00:48:18.128179 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 00:48:18.129282 16051 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:48:18.253382 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 15851
I20260810 00:48:18.282032 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 15269
I20260810 00:48:18.283352 15367 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:48:18.392874 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 15269
I20260810 00:48:18.426640 28696 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 00:48:18.771386 28696 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 00:48:18.771620 28696 mini_ranger.cc:66] Stopping Ranger...
I20260810 00:48:19.116659 28696 mini_ranger.cc:68] Stopped Ranger
2026-08-10 00:48:19.116 UTC [13808] LOG: received smart shutdown request
2026-08-10 00:48:19.119 UTC [13808] LOG: background worker "logical replication launcher" (PID 13822) exited with exit code 1
2026-08-10 00:48:19.120 UTC [13813] LOG: shutting down
2026-08-10 00:48:19.261 UTC [13808] LOG: database system is shut down
2026-08-10T00:48:19Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53310 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:48:19 dist-test-slave-1nl6 krb5kdc[16159](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:48:19 dist-test-slave-1nl6 krb5kdc[16159](info): set up 2 sockets
Aug 10 00:48:19 dist-test-slave-1nl6 krb5kdc[16159](info): commencing operation
krb5kdc: starting...
W20260810 00:48:20.699569 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.249s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:48:20 dist-test-slave-1nl6 krb5kdc[16159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:48:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:48:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.6.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
I20260810 00:48:20.940577 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:34753
--webserver_interface=127.28.6.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:42299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:34753,127.28.6.61:40475,127.28.6.60:41333
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:48:21.273568 16179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:48:21.273833 16179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:48:21.273918 16179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:48:21.280735 16179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:48:21.280834 16179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:48:21.280889 16179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:48:21.280943 16179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:48:21.281037 16179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:48:21.290318 16179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:42299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:34753,127.28.6.61:40475,127.28.6.60:41333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:34753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:48:21.291517 16179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:48:21.292749 16179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:48:21.304318 16190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:21.304270 16185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:21.304191 16186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:48:21.305570 16179 server_base.cc:1034] running on GCE node
I20260810 00:48:21.306329 16179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:48:21.307703 16179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:48:21.308939 16179 hybrid_clock.cc:648] HybridClock initialized: now 1786322901308897 us; error 57 us; skew 500 ppm
I20260810 00:48:21.309281 16179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:48:21 dist-test-slave-1nl6 krb5kdc[16159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322901, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:48:21.312427 16179 init.cc:377] Logged in from keytab as kudu/127.28.6.62@KRBTEST.COM (short username kudu)
I20260810 00:48:21.313916 16179 webserver.cc:460] Webserver started at http://127.28.6.62:33549/ using document root <none> and password file <none>
I20260810 00:48:21.314612 16179 fs_manager.cc:359] Metadata directory not provided
I20260810 00:48:21.314790 16179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:48:21.315131 16179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:48:21.317945 16179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "ea1e291e18b04f43af3b885eaead9c51"
format_stamp: "Formatted at 2026-08-10 00:48:21 on dist-test-slave-1nl6"
server_key: "c8a8e8b820b228717f53b6bf5912c172"
server_key_iv: "b3a4593d045ec1c0916b66f0c86981e3"
server_key_version: "encryptionkey@0"
I20260810 00:48:21.318743 16179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "ea1e291e18b04f43af3b885eaead9c51"
format_stamp: "Formatted at 2026-08-10 00:48:21 on dist-test-slave-1nl6"
server_key: "c8a8e8b820b228717f53b6bf5912c172"
server_key_iv: "b3a4593d045ec1c0916b66f0c86981e3"
server_key_version: "encryptionkey@0"
I20260810 00:48:21.323643 16179 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260810 00:48:21.326886 16201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:21.328176 16179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 00:48:21.328368 16179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "ea1e291e18b04f43af3b885eaead9c51"
format_stamp: "Formatted at 2026-08-10 00:48:21 on dist-test-slave-1nl6"
server_key: "c8a8e8b820b228717f53b6bf5912c172"
server_key_iv: "b3a4593d045ec1c0916b66f0c86981e3"
server_key_version: "encryptionkey@0"
I20260810 00:48:21.328557 16179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:48:21.357223 16179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:48:21.361157 16179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:48:21.361454 16179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:48:21.379603 16179 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:34753
I20260810 00:48:21.379645 16280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:34753 every 8 connection(s)
I20260810 00:48:21.380877 16179 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:48:21.382858 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 16179
I20260810 00:48:21.383128 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:48:21.383961 28696 external_mini_cluster.cc:1428] Setting key e282c2920a98025b55799c957338eb58
I20260810 00:48:21.385109 16283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:48:21.394865 16283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34753 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 00:48:21 dist-test-slave-1nl6 krb5kdc[16159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322901, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.6.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.61@KRBTEST.COM" created.
I20260810 00:48:21.419855 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:21.397914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:34753 (local address 127.0.0.1:52710)
0810 00:48:21.399222 (+ 1308us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:21.399270 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:21.401979 (+ 2709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:21.402834 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:21.402844 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:21.403674 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:48:21.404171 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:21.404197 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:21.405714 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:21.405723 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:21.407673 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:21.407693 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:21.408656 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:21.408667 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:21.408910 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:21.409608 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:48:21.409641 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:48:21.412370 (+ 2729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:48:21.415757 (+ 3387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:21.415773 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:21.415787 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:21.416414 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:21.416832 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:21.416840 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:21.416847 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:21.417060 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:21.417786 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:48:21.417797 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:48:21.418077 (+ 280us) client_negotiation.cc:769] Sending connection context
0810 00:48:21.418357 (+ 280us) client_negotiation.cc:240] Negotiation successful
0810 00:48:21.418950 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1034,"mutex_wait_us":49,"spinlock_wait_cycles":12672,"thread_start_us":812,"threads_started":1}
I20260810 00:48:21.419893 16287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:21.397916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52710 (local address 127.28.6.62:34753)
0810 00:48:21.398848 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:21.398864 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:21.399418 (+ 554us) server_negotiation.cc:407] Connection header received
0810 00:48:21.402378 (+ 2960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:21.402383 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:21.402586 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:21.402734 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:21.404372 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:21.405513 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:21.407832 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:21.408386 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:21.408510 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:21.412615 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:21.412624 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:21.412635 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:21.412663 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:21.415604 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:21.416554 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:21.416560 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:21.416562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:21.416698 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:21.417172 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:21.417175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:21.417177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:21.417625 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:21.417792 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:21.418493 (+ 701us) server_negotiation.cc:299] Negotiation successful
0810 00:48:21.418749 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":662,"thread_start_us":429,"threads_started":1}
I20260810 00:48:21.422745 16283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.6.61" port: 40475 } has no permanent_uuid. Determining permanent_uuid...
I20260810 00:48:21.423777 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:21.423425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:40475 (local address 127.0.0.1:44256)
0810 00:48:21.423580 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:21.423679 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.61:40475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260810 00:48:21.424083 16202 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.6.61:40475: connect: Connection refused (error 111)
I20260810 00:48:21.425810 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:21.425367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:40475 (local address 127.0.0.1:44262)
0810 00:48:21.425532 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:21.425634 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.61:40475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 00:48:21.426055 16283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.61:40475: Network error: Client connection negotiation failed: client connection to 127.28.6.61:40475: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.61.keytab.
Entry for principal kudu/127.28.6.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.61.keytab.
I20260810 00:48:21.446115 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.61:40475
--webserver_interface=127.28.6.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.61.keytab
--principal=kudu/127.28.6.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:42299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:34753,127.28.6.61:40475,127.28.6.60:41333
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 00:48:21.475493 16283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.61" port: 40475 } attempt: 1
I20260810 00:48:21.476713 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:21.476244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:40475 (local address 127.0.0.1:44264)
0810 00:48:21.476442 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:21.476575 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.61:40475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260810 00:48:21.477730 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:21.477417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:40475 (local address 127.0.0.1:44278)
0810 00:48:21.477542 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:21.477604 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.61:40475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 00:48:21.477953 16283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.61:40475: Network error: Client connection negotiation failed: client connection to 127.28.6.61:40475: connect: Connection refused (error 111)
I20260810 00:48:21.546223 16283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.61" port: 40475 } attempt: 2
I20260810 00:48:21.547540 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:21.547035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:40475 (local address 127.0.0.1:44290)
0810 00:48:21.547307 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:21.547407 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.61:40475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260810 00:48:21.548694 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:21.548343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:40475 (local address 127.0.0.1:44298)
0810 00:48:21.548497 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:21.548566 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.61:40475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 00:48:21.548987 16283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.61:40475: Network error: Client connection negotiation failed: client connection to 127.28.6.61:40475: connect: Connection refused (error 111)
I20260810 00:48:21.640301 16283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.61" port: 40475 } attempt: 3
I20260810 00:48:21.641463 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:21.640985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:40475 (local address 127.0.0.1:44300)
0810 00:48:21.641207 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:21.641309 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.61:40475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260810 00:48:21.642479 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:21.642117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:40475 (local address 127.0.0.1:44308)
0810 00:48:21.642273 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:21.642333 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.61:40475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 00:48:21.642709 16283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.61:40475: Network error: Client connection negotiation failed: client connection to 127.28.6.61:40475: connect: Connection refused (error 111)
W20260810 00:48:21.776298 16291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:48:21.776567 16291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:48:21.776684 16291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:48:21.783856 16291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:48:21.783984 16291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:48:21.784083 16291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:48:21.784195 16291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:48:21.784276 16291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:48:21.785995 16283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.61" port: 40475 } attempt: 4
I20260810 00:48:21.787304 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:21.786719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:40475 (local address 127.0.0.1:44324)
0810 00:48:21.787062 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:21.787164 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.61:40475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194}
I20260810 00:48:21.788307 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:21.787945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:40475 (local address 127.0.0.1:44326)
0810 00:48:21.788095 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:21.788179 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.61:40475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 00:48:21.788573 16283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.61:40475: Network error: Client connection negotiation failed: client connection to 127.28.6.61:40475: connect: Connection refused (error 111)
I20260810 00:48:21.794157 16291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:42299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:34753,127.28.6.61:40475,127.28.6.60:41333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.61.keytab
--principal=kudu/127.28.6.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.61:40475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.6.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:48:21.795531 16291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:48:21.796836 16291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:48:21.808451 16302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:21.808544 16305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:21.808499 16303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:48:21.809547 16291 server_base.cc:1034] running on GCE node
I20260810 00:48:21.810303 16291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:48:21.811745 16291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:48:21.812945 16291 hybrid_clock.cc:648] HybridClock initialized: now 1786322901812893 us; error 50 us; skew 500 ppm
I20260810 00:48:21.813290 16291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:48:21 dist-test-slave-1nl6 krb5kdc[16159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322901, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:48:21.816725 16291 init.cc:377] Logged in from keytab as kudu/127.28.6.61@KRBTEST.COM (short username kudu)
I20260810 00:48:21.818629 16291 webserver.cc:460] Webserver started at http://127.28.6.61:38009/ using document root <none> and password file <none>
I20260810 00:48:21.819572 16291 fs_manager.cc:359] Metadata directory not provided
I20260810 00:48:21.819727 16291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:48:21.820055 16291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:48:21.824173 16291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-1/data/instance:
uuid: "8904933c420149658a2a111ff8d2b6ae"
format_stamp: "Formatted at 2026-08-10 00:48:21 on dist-test-slave-1nl6"
server_key: "fb6295c30a8b41ce30834fe656aa9067"
server_key_iv: "62f6a69b2ea77a53442272e546cd49a9"
server_key_version: "encryptionkey@0"
I20260810 00:48:21.825109 16291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-1/wal/instance:
uuid: "8904933c420149658a2a111ff8d2b6ae"
format_stamp: "Formatted at 2026-08-10 00:48:21 on dist-test-slave-1nl6"
server_key: "fb6295c30a8b41ce30834fe656aa9067"
server_key_iv: "62f6a69b2ea77a53442272e546cd49a9"
server_key_version: "encryptionkey@0"
I20260810 00:48:21.829982 16291 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 00:48:21.833391 16316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:21.834656 16291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 00:48:21.834793 16291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-1/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-1/wal
uuid: "8904933c420149658a2a111ff8d2b6ae"
format_stamp: "Formatted at 2026-08-10 00:48:21 on dist-test-slave-1nl6"
server_key: "fb6295c30a8b41ce30834fe656aa9067"
server_key_iv: "62f6a69b2ea77a53442272e546cd49a9"
server_key_version: "encryptionkey@0"
I20260810 00:48:21.835016 16291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:48:21.853297 16291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:48:21.857297 16291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:48:21.857515 16291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:48:21.875561 16291 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.61:40475
I20260810 00:48:21.875605 16394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.61:40475 every 8 connection(s)
I20260810 00:48:21.876883 16291 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-1/data/info.pb
I20260810 00:48:21.879964 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 16291
I20260810 00:48:21.880151 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-1/wal/instance
I20260810 00:48:21.880523 16395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:48:21.880687 28696 external_mini_cluster.cc:1428] Setting key d148bfe920a16be41aa965cc7c80ba4d
I20260810 00:48:21.886822 16395 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34753 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 00:48:21 dist-test-slave-1nl6 krb5kdc[16159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322901, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.61@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.6.60@KRBTEST.COM; defaulting to no policy
I20260810 00:48:21.907949 16287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:21.889430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52726 (local address 127.28.6.62:34753)
0810 00:48:21.889610 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:21.889614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:21.890988 (+ 1374us) server_negotiation.cc:407] Connection header received
0810 00:48:21.893132 (+ 2144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:21.893136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:21.893205 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:21.893308 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:21.895332 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:21.895976 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:21.898437 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:21.898841 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:21.899068 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:21.902367 (+ 3299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:21.902372 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:21.902375 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:21.902406 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:21.904995 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:21.905728 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:21.905736 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:21.905741 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:21.905868 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:21.906292 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:21.906297 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:21.906300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:21.906572 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:21.906752 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:21.907425 (+ 673us) server_negotiation.cc:299] Negotiation successful
0810 00:48:21.907641 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 00:48:21.908885 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:21.889888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:34753 (local address 127.0.0.1:52726)
0810 00:48:21.890777 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:21.890828 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:21.892890 (+ 2062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:21.893553 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:21.893567 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:21.894341 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:48:21.895126 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:21.895153 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:21.896144 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:21.896152 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:21.898281 (+ 2129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:21.898300 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:21.899159 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:21.899167 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:21.899385 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:21.900073 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:48:21.900094 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:48:21.902145 (+ 2051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:48:21.905142 (+ 2997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:21.905151 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:21.905165 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:21.905604 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:21.905981 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:21.905986 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:21.905989 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:21.906163 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:21.906715 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:48:21.906732 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:48:21.907092 (+ 360us) client_negotiation.cc:769] Sending connection context
0810 00:48:21.907340 (+ 248us) client_negotiation.cc:240] Negotiation successful
0810 00:48:21.907792 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":339,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.60@KRBTEST.COM" created.
I20260810 00:48:21.910625 16395 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.6.61" port: 40475 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 00:48:21 dist-test-slave-1nl6 krb5kdc[16159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322901, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.61@KRBTEST.COM for kudu/127.28.6.61@KRBTEST.COM
I20260810 00:48:21.926465 16404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:21.911794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44334 (local address 127.28.6.61:40475)
0810 00:48:21.912589 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:21.912599 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:21.912631 (+ 32us) server_negotiation.cc:407] Connection header received
0810 00:48:21.912688 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:21.912697 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:21.912911 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:21.913058 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:21.913763 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:21.915599 (+ 1836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:21.916790 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:21.917697 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:21.917852 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:21.920783 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:21.920794 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:21.920805 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:21.920837 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:21.923524 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:21.924352 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:21.924358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:21.924360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:21.924469 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:21.924879 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:21.924882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:21.924884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:21.925208 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:21.925379 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:21.925846 (+ 467us) server_negotiation.cc:299] Negotiation successful
0810 00:48:21.926168 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":133,"server-negotiator.queue_time_us":604,"thread_start_us":437,"threads_started":1}
I20260810 00:48:21.926542 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:21.911487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:40475 (local address 127.0.0.1:44334)
0810 00:48:21.911682 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:21.911710 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:21.911817 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:21.913082 (+ 1265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:21.913085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:21.913455 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:48:21.913582 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:21.913589 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:21.915780 (+ 2191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:21.915784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:21.916646 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:21.916657 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:21.917860 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:21.917864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:21.917993 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:21.918560 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:48:21.918576 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:48:21.920581 (+ 2005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:48:21.923658 (+ 3077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:21.923666 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:21.923673 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:21.924224 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:21.924587 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:21.924595 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:21.924601 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:21.924778 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:21.925360 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:48:21.925367 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:48:21.925549 (+ 182us) client_negotiation.cc:769] Sending connection context
0810 00:48:21.925736 (+ 187us) client_negotiation.cc:240] Negotiation successful
0810 00:48:21.926054 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260810 00:48:21.928846 16395 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.6.60" port: 41333 } has no permanent_uuid. Determining permanent_uuid...
I20260810 00:48:21.929865 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:21.929469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:41333 (local address 127.0.0.1:48808)
0810 00:48:21.929625 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:21.929739 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:41333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260810 00:48:21.930189 16322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.6.60:41333: connect: Connection refused (error 111)
I20260810 00:48:21.931789 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:21.931498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:41333 (local address 127.0.0.1:48810)
0810 00:48:21.931628 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:21.931700 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:41333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 00:48:21.932199 16395 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.60:41333: Network error: Client connection negotiation failed: client connection to 127.28.6.60:41333: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.60.keytab.
Entry for principal kudu/127.28.6.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.60.keytab.
I20260810 00:48:21.942414 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.60:41333
--webserver_interface=127.28.6.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.60.keytab
--principal=kudu/127.28.6.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:42299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:34753,127.28.6.61:40475,127.28.6.60:41333
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 00:48:21.981510 16395 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.60" port: 41333 } attempt: 1
I20260810 00:48:21.982707 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:21.982264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:41333 (local address 127.0.0.1:48820)
0810 00:48:21.982454 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:21.982567 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:41333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260810 00:48:21.983724 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:21.983398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:41333 (local address 127.0.0.1:48824)
0810 00:48:21.983540 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:21.983624 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:41333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 00:48:21.983934 16395 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.60:41333: Network error: Client connection negotiation failed: client connection to 127.28.6.60:41333: connect: Connection refused (error 111)
I20260810 00:48:22.052143 16395 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.60" port: 41333 } attempt: 2
I20260810 00:48:22.053288 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:22.052827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:41333 (local address 127.0.0.1:48834)
0810 00:48:22.053025 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:22.053131 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:41333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260810 00:48:22.054375 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:22.054050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:41333 (local address 127.0.0.1:48842)
0810 00:48:22.054201 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:22.054270 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:41333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 00:48:22.054581 16395 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.60:41333: Network error: Client connection negotiation failed: client connection to 127.28.6.60:41333: connect: Connection refused (error 111)
I20260810 00:48:22.087848 16283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.61" port: 40475 } attempt: 5
Aug 10 00:48:22 dist-test-slave-1nl6 krb5kdc[16159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322901, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for kudu/127.28.6.61@KRBTEST.COM
I20260810 00:48:22.101240 16404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:22.088711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44342 (local address 127.28.6.61:40475)
0810 00:48:22.088928 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:22.088936 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:22.088959 (+ 23us) server_negotiation.cc:407] Connection header received
0810 00:48:22.089185 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:22.089190 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:22.089255 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:22.089365 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:22.090183 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:22.090862 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:22.092040 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:22.092539 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:22.092678 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:22.095895 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:22.095900 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:22.095903 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:22.095934 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:22.098613 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:22.099368 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:22.099378 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:22.099381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:22.099509 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:22.099893 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:22.099899 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:22.099902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:22.100181 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:22.100348 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:22.100717 (+ 369us) server_negotiation.cc:299] Negotiation successful
0810 00:48:22.100930 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260810 00:48:22.101254 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:22.088627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:40475 (local address 127.0.0.1:44342)
0810 00:48:22.088860 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:22.088880 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:22.089031 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:22.089405 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:22.089411 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:22.089814 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:48:22.089994 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:22.090003 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:22.091035 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:22.091043 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:22.091892 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:22.091903 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:22.092690 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:22.092695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:22.092910 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:22.093603 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:48:22.093620 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:48:22.095707 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:48:22.098759 (+ 3052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:22.098764 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:22.098769 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:22.099226 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:22.099624 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:22.099629 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:22.099633 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:22.099783 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:22.100305 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:48:22.100310 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:48:22.100475 (+ 165us) client_negotiation.cc:769] Sending connection context
0810 00:48:22.100614 (+ 139us) client_negotiation.cc:240] Negotiation successful
0810 00:48:22.100879 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260810 00:48:22.102649 16283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.6.60" port: 41333 } has no permanent_uuid. Determining permanent_uuid...
I20260810 00:48:22.103657 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:22.103288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:41333 (local address 127.0.0.1:48852)
0810 00:48:22.103444 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:22.103522 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:41333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 00:48:22.104535 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:22.104210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:41333 (local address 127.0.0.1:48862)
0810 00:48:22.104380 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:22.104442 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:41333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 00:48:22.104758 16283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.60:41333: Network error: Client connection negotiation failed: client connection to 127.28.6.60:41333: connect: Connection refused (error 111)
I20260810 00:48:22.145781 16395 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.60" port: 41333 } attempt: 3
I20260810 00:48:22.146860 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:22.146404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:41333 (local address 127.0.0.1:48866)
0810 00:48:22.146619 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:22.146705 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:41333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260810 00:48:22.148007 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:22.147594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:41333 (local address 127.0.0.1:48876)
0810 00:48:22.147810 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:22.147897 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:41333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260810 00:48:22.148227 16395 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.60:41333: Network error: Client connection negotiation failed: client connection to 127.28.6.60:41333: connect: Connection refused (error 111)
I20260810 00:48:22.155956 16283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.60" port: 41333 } attempt: 1
I20260810 00:48:22.157006 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:22.156472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:41333 (local address 127.0.0.1:48882)
0810 00:48:22.156709 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:22.156810 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:41333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260810 00:48:22.158105 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:22.157715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:41333 (local address 127.0.0.1:48892)
0810 00:48:22.157868 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:22.157957 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:41333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 00:48:22.158365 16283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.60:41333: Network error: Client connection negotiation failed: client connection to 127.28.6.60:41333: connect: Connection refused (error 111)
I20260810 00:48:22.226634 16283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.60" port: 41333 } attempt: 2
I20260810 00:48:22.227941 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:22.227361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:41333 (local address 127.0.0.1:48908)
0810 00:48:22.227606 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:22.227728 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:41333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260810 00:48:22.229089 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:22.228639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:41333 (local address 127.0.0.1:48922)
0810 00:48:22.228851 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:22.228943 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:41333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260810 00:48:22.229316 16283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.60:41333: Network error: Client connection negotiation failed: client connection to 127.28.6.60:41333: connect: Connection refused (error 111)
W20260810 00:48:22.277690 16407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:48:22.277941 16407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:48:22.278025 16407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:48:22.284935 16407 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:48:22.285029 16407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:48:22.285074 16407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:48:22.285131 16407 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:48:22.285174 16407 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:48:22.291445 16395 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.60" port: 41333 } attempt: 4
I20260810 00:48:22.292531 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:22.292038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:41333 (local address 127.0.0.1:48924)
0810 00:48:22.292306 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:22.292387 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:41333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260810 00:48:22.293515 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:22.293204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:41333 (local address 127.0.0.1:48934)
0810 00:48:22.293357 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:22.293418 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:41333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260810 00:48:22.293771 16395 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.60:41333: Network error: Client connection negotiation failed: client connection to 127.28.6.60:41333: connect: Connection refused (error 111)
I20260810 00:48:22.294721 16407 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:42299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:34753,127.28.6.61:40475,127.28.6.60:41333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.60.keytab
--principal=kudu/127.28.6.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.60:41333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.6.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:48:22.295949 16407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:48:22.297214 16407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:48:22.308830 16415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:22.309482 16421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:22.309836 16419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:48:22.309978 16407 server_base.cc:1034] running on GCE node
I20260810 00:48:22.310760 16407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:48:22.312191 16407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:48:22.313436 16407 hybrid_clock.cc:648] HybridClock initialized: now 1786322902313378 us; error 48 us; skew 500 ppm
I20260810 00:48:22.313768 16407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:48:22 dist-test-slave-1nl6 krb5kdc[16159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322902, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:48:22.316913 16407 init.cc:377] Logged in from keytab as kudu/127.28.6.60@KRBTEST.COM (short username kudu)
I20260810 00:48:22.318257 16407 webserver.cc:460] Webserver started at http://127.28.6.60:39073/ using document root <none> and password file <none>
I20260810 00:48:22.318938 16407 fs_manager.cc:359] Metadata directory not provided
I20260810 00:48:22.319094 16407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:48:22.319396 16407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:48:22.322157 16407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-2/data/instance:
uuid: "5d2e4f0de55f4f13a9e122ab0eb518dc"
format_stamp: "Formatted at 2026-08-10 00:48:22 on dist-test-slave-1nl6"
server_key: "36bac8c88d39f831b84859c5ea8475d0"
server_key_iv: "4c37d6ca388b17063875e100e5a6b8d3"
server_key_version: "encryptionkey@0"
I20260810 00:48:22.322993 16407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-2/wal/instance:
uuid: "5d2e4f0de55f4f13a9e122ab0eb518dc"
format_stamp: "Formatted at 2026-08-10 00:48:22 on dist-test-slave-1nl6"
server_key: "36bac8c88d39f831b84859c5ea8475d0"
server_key_iv: "4c37d6ca388b17063875e100e5a6b8d3"
server_key_version: "encryptionkey@0"
I20260810 00:48:22.327646 16407 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 00:48:22.330799 16434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:22.332188 16407 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 00:48:22.332388 16407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-2/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-2/wal
uuid: "5d2e4f0de55f4f13a9e122ab0eb518dc"
format_stamp: "Formatted at 2026-08-10 00:48:22 on dist-test-slave-1nl6"
server_key: "36bac8c88d39f831b84859c5ea8475d0"
server_key_iv: "4c37d6ca388b17063875e100e5a6b8d3"
server_key_version: "encryptionkey@0"
I20260810 00:48:22.332644 16407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:48:22.335542 16283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.60" port: 41333 } attempt: 3
I20260810 00:48:22.336678 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:22.336117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:41333 (local address 127.0.0.1:48944)
0810 00:48:22.336364 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:22.336472 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:41333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260810 00:48:22.337805 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:22.337321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:41333 (local address 127.0.0.1:48950)
0810 00:48:22.337512 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:22.337623 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:41333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260810 00:48:22.338061 16283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.60:41333: Network error: Client connection negotiation failed: client connection to 127.28.6.60:41333: connect: Connection refused (error 111)
I20260810 00:48:22.345281 16407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:48:22.349669 16407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:48:22.349982 16407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:48:22.368630 16407 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.60:41333
I20260810 00:48:22.368664 16513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.60:41333 every 8 connection(s)
I20260810 00:48:22.369971 16407 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-2/data/info.pb
I20260810 00:48:22.373558 16514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:48:22.375427 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 16407
I20260810 00:48:22.375690 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786322687226232-28696-0/minicluster-data/master-2/wal/instance
I20260810 00:48:22.376276 28696 external_mini_cluster.cc:1428] Setting key 1c90e2e2a713d21b926273efc0ae5ffa
I20260810 00:48:22.381717 16514 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34753 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 00:48:22 dist-test-slave-1nl6 krb5kdc[16159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.60@KRBTEST.COM
Aug 10 00:48:22 dist-test-slave-1nl6 krb5kdc[16159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322902, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.60@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:48:22.400012 16287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:22.384676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52730 (local address 127.28.6.62:34753)
0810 00:48:22.384852 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:22.384858 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:22.385849 (+ 991us) server_negotiation.cc:407] Connection header received
0810 00:48:22.386157 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:22.386165 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:22.386246 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:22.386368 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:22.387582 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:22.388637 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:22.390371 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:22.390800 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:22.390953 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:22.394274 (+ 3321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:22.394279 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:22.394283 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:22.394312 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:22.397000 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:22.397779 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:22.397785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:22.397789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:22.397928 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:22.398372 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:22.398378 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:22.398381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:22.398620 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:22.398793 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:22.399466 (+ 673us) server_negotiation.cc:299] Negotiation successful
0810 00:48:22.399669 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 00:48:22.401021 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:22.384784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:34753 (local address 127.0.0.1:52730)
0810 00:48:22.385710 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:22.385762 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:22.385987 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:22.386438 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:22.386445 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:22.387122 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:48:22.387418 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:22.387428 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:22.388844 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:22.388850 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:22.390208 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:22.390220 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:22.391055 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:22.391064 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:22.391218 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:22.391848 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:48:22.391869 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:48:22.394081 (+ 2212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:48:22.397130 (+ 3049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:22.397143 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:22.397153 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:22.397658 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:22.398040 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:22.398045 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:22.398048 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:22.398217 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:22.398748 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:48:22.398756 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:48:22.399107 (+ 351us) client_negotiation.cc:769] Sending connection context
0810 00:48:22.399359 (+ 252us) client_negotiation.cc:240] Negotiation successful
0810 00:48:22.399790 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":456,"threads_started":1}
I20260810 00:48:22.401101 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:22.378255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48962 (local address 127.28.6.60:41333)
0810 00:48:22.379297 (+ 1042us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:22.379309 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:22.379362 (+ 53us) server_negotiation.cc:407] Connection header received
0810 00:48:22.381111 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:22.381133 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:22.381530 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:22.381961 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:22.383769 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:22.385976 (+ 2207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:22.388078 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:22.389079 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:22.389318 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:22.394214 (+ 4896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:22.394223 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:22.394234 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:22.394266 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:22.397593 (+ 3327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:22.398396 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:22.398402 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:22.398405 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:22.398542 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:22.398997 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:22.399001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:22.399004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:22.399341 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:22.399688 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:22.400053 (+ 365us) server_negotiation.cc:299] Negotiation successful
0810 00:48:22.400472 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":444,"threads_started":1}
I20260810 00:48:22.402489 16514 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.6.61" port: 40475 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 00:48:22 dist-test-slave-1nl6 krb5kdc[16159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322902, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.60@KRBTEST.COM for kudu/127.28.6.61@KRBTEST.COM
Aug 10 00:48:22 dist-test-slave-1nl6 krb5kdc[16159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:48:22.415874 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:22.403219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:40475 (local address 127.0.0.1:44358)
0810 00:48:22.403367 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:22.403385 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:22.403472 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:22.403812 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:22.403817 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:22.404222 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:48:22.404387 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:22.404404 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:22.405393 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:22.405398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:22.406234 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:22.406245 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:22.406953 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:22.406959 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:22.407157 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:22.407669 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:48:22.407683 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:48:22.409886 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:48:22.413295 (+ 3409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:22.413301 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:22.413308 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:22.413839 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:22.414204 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:22.414207 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:22.414209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:22.414359 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:22.414885 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:48:22.414890 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:48:22.415100 (+ 210us) client_negotiation.cc:769] Sending connection context
0810 00:48:22.415248 (+ 148us) client_negotiation.cc:240] Negotiation successful
0810 00:48:22.415498 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 00:48:22.415895 16404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:22.403266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44358 (local address 127.28.6.61:40475)
0810 00:48:22.403542 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:22.403550 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:22.403567 (+ 17us) server_negotiation.cc:407] Connection header received
0810 00:48:22.403630 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:22.403634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:22.403687 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:22.403791 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:22.404597 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:22.405248 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:22.406373 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:22.406780 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:22.406909 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:22.410088 (+ 3179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:22.410096 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:22.410101 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:22.410144 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:22.413163 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:22.413952 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:22.413956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:22.413958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:22.414095 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:22.414489 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:22.414494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:22.414497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:22.414761 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:22.414948 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:22.415336 (+ 388us) server_negotiation.cc:299] Negotiation successful
0810 00:48:22.415530 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260810 00:48:22.417224 16514 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.6.60" port: 41333 } has no permanent_uuid. Determining permanent_uuid...
I20260810 00:48:22.419859 16287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:22.404440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52734 (local address 127.28.6.62:34753)
0810 00:48:22.404604 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:22.404612 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:22.404632 (+ 20us) server_negotiation.cc:407] Connection header received
0810 00:48:22.404843 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:22.404848 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:22.404920 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:22.405026 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:22.406260 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:22.407113 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:22.408766 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:22.409208 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:22.409312 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:22.414265 (+ 4953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:22.414269 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:22.414272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:22.414308 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:22.417150 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:22.417888 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:22.417893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:22.417898 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:22.418057 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:22.418455 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:22.418460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:22.418464 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:22.418736 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:22.419048 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:22.419333 (+ 285us) server_negotiation.cc:299] Negotiation successful
0810 00:48:22.419537 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Aug 10 00:48:22 dist-test-slave-1nl6 krb5kdc[16159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322902, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.60@KRBTEST.COM for kudu/127.28.6.60@KRBTEST.COM
Aug 10 00:48:22 dist-test-slave-1nl6 krb5kdc[16159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.61@KRBTEST.COM
I20260810 00:48:22.429877 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:22.418117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48978 (local address 127.28.6.60:41333)
0810 00:48:22.418282 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:22.418287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:22.418308 (+ 21us) server_negotiation.cc:407] Connection header received
0810 00:48:22.418339 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:22.418342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:22.418385 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:22.418482 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:22.419197 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:22.419990 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:22.421474 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:22.421906 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:22.421974 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:22.424783 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:22.424787 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:22.424790 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:22.424815 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:22.427491 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:22.428108 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:22.428114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:22.428118 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:22.428229 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:22.428634 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:22.428638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:22.428640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:22.428922 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:22.429091 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:22.429397 (+ 306us) server_negotiation.cc:299] Negotiation successful
0810 00:48:22.429640 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":65}
I20260810 00:48:22.429904 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:22.417875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:41333 (local address 127.0.0.1:48978)
0810 00:48:22.418062 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:22.418081 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:22.418175 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:22.418499 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:22.418503 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:22.418870 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:48:22.419054 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:22.419062 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:22.420139 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:22.420143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:22.421334 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:22.421350 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:22.422112 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:22.422115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:22.422260 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:22.422777 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:48:22.422793 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:48:22.424611 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:48:22.427633 (+ 3022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:22.427639 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:22.427642 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:22.427996 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:22.428344 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:22.428349 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:22.428353 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:22.428515 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:22.429075 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:48:22.429080 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:48:22.429198 (+ 118us) client_negotiation.cc:769] Sending connection context
0810 00:48:22.429315 (+ 117us) client_negotiation.cc:240] Negotiation successful
0810 00:48:22.429520 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 00:48:22.435690 16514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d2e4f0de55f4f13a9e122ab0eb518dc: Bootstrap starting.
I20260810 00:48:22.436293 16404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:22.421507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44372 (local address 127.28.6.61:40475)
0810 00:48:22.421671 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:22.421679 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:22.421696 (+ 17us) server_negotiation.cc:407] Connection header received
0810 00:48:22.421870 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:22.421875 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:22.421938 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:22.422038 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:22.423304 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:22.423932 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:22.425254 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:22.425701 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:22.425838 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:22.430764 (+ 4926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:22.430770 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:22.430773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:22.430809 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:22.433667 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:22.434453 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:22.434457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:22.434459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:22.434579 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:22.435005 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:22.435008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:22.435010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:22.435260 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:22.435454 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:22.435845 (+ 391us) server_negotiation.cc:299] Negotiation successful
0810 00:48:22.436009 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 00:48:22.439664 16514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d2e4f0de55f4f13a9e122ab0eb518dc: Neither blocks nor log segments found. Creating new log.
I20260810 00:48:22.440991 16514 log.cc:824] T 00000000000000000000000000000000 P 5d2e4f0de55f4f13a9e122ab0eb518dc: Log is configured to *not* fsync() on all Append() calls
I20260810 00:48:22.441471 28696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 00:48:22.443679 16514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d2e4f0de55f4f13a9e122ab0eb518dc: No bootstrap required, opened a new log
I20260810 00:48:22.454876 16514 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d2e4f0de55f4f13a9e122ab0eb518dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1e291e18b04f43af3b885eaead9c51" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34753 } } peers { permanent_uuid: "8904933c420149658a2a111ff8d2b6ae" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 40475 } } peers { permanent_uuid: "5d2e4f0de55f4f13a9e122ab0eb518dc" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 41333 } }
I20260810 00:48:22.455325 16514 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d2e4f0de55f4f13a9e122ab0eb518dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:48:22.455459 16514 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d2e4f0de55f4f13a9e122ab0eb518dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d2e4f0de55f4f13a9e122ab0eb518dc, State: Initialized, Role: FOLLOWER
I20260810 00:48:22.456210 16514 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d2e4f0de55f4f13a9e122ab0eb518dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1e291e18b04f43af3b885eaead9c51" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34753 } } peers { permanent_uuid: "8904933c420149658a2a111ff8d2b6ae" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 40475 } } peers { permanent_uuid: "5d2e4f0de55f4f13a9e122ab0eb518dc" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 41333 } }
I20260810 00:48:22.468195 16287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:22.446403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52746 (local address 127.28.6.62:34753)
0810 00:48:22.446573 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:22.446580 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:22.447431 (+ 851us) server_negotiation.cc:407] Connection header received
0810 00:48:22.447740 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:22.447749 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:22.447850 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:22.447934 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:22.450866 (+ 2932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:22.452367 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:22.455679 (+ 3312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:22.456370 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:22.456471 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:22.460502 (+ 4031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:22.460508 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:22.460513 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:22.460553 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:22.464566 (+ 4013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:22.465544 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:22.465550 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:22.465554 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:22.465735 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:22.466228 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:22.466233 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:22.466237 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:22.466584 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:22.466856 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:22.467518 (+ 662us) server_negotiation.cc:299] Negotiation successful
0810 00:48:22.467769 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 00:48:22.470788 16404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:22.447039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44380 (local address 127.28.6.61:40475)
0810 00:48:22.447187 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:22.447192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:22.450557 (+ 3365us) server_negotiation.cc:407] Connection header received
0810 00:48:22.450736 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:22.450742 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:22.450815 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:22.450929 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:22.453305 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:22.454335 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:22.456555 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:22.457380 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:22.457507 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:22.462160 (+ 4653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:22.462170 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:22.462177 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:22.462227 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:22.466691 (+ 4464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:22.467806 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:22.467815 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:22.467821 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:22.467997 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:22.468544 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:22.468551 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:22.468557 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:22.469012 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:22.469323 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:22.469977 (+ 654us) server_negotiation.cc:299] Negotiation successful
0810 00:48:22.470294 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 00:48:22.471311 16532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d2e4f0de55f4f13a9e122ab0eb518dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1e291e18b04f43af3b885eaead9c51" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34753 } } peers { permanent_uuid: "8904933c420149658a2a111ff8d2b6ae" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 40475 } } peers { permanent_uuid: "5d2e4f0de55f4f13a9e122ab0eb518dc" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 41333 } } }
I20260810 00:48:22.471772 16532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d2e4f0de55f4f13a9e122ab0eb518dc [sys.catalog]: This master's current role is: FOLLOWER
I20260810 00:48:22.472810 16514 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d2e4f0de55f4f13a9e122ab0eb518dc [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:48:22.473201 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:22.447568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48994 (local address 127.28.6.60:41333)
0810 00:48:22.447797 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:22.447805 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:22.455542 (+ 7737us) server_negotiation.cc:407] Connection header received
0810 00:48:22.455763 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:22.455769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:22.455832 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:22.455895 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:22.458348 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:22.459519 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:22.461819 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:22.462630 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:22.462728 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:22.466162 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:22.466170 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:22.466177 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:22.466218 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:22.469780 (+ 3562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:22.470695 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:22.470703 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:22.470708 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:22.470917 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:22.471447 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:22.471454 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:22.471459 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:22.471844 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:22.472109 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:22.472522 (+ 413us) server_negotiation.cc:299] Negotiation successful
0810 00:48:22.472733 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260810 00:48:22.483083 16546 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5d2e4f0de55f4f13a9e122ab0eb518dc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 00:48:22.483230 16546 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 00:48:22.479414 16407 master_runner.cc:186] Error getting master registration for 127.28.6.62:34753: OK, instance_id { permanent_uuid: "ea1e291e18b04f43af3b885eaead9c51" instance_seqno: 1786322901374803 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 00:48:22.488782 16407 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 00:48:22.515282 16283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.60" port: 41333 } attempt: 4
Aug 10 00:48:22 dist-test-slave-1nl6 krb5kdc[16159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322901, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for kudu/127.28.6.60@KRBTEST.COM
I20260810 00:48:22.529356 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:22.515890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49004 (local address 127.28.6.60:41333)
0810 00:48:22.516058 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:22.516065 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:22.516085 (+ 20us) server_negotiation.cc:407] Connection header received
0810 00:48:22.516319 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:22.516327 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:22.516434 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:22.516529 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:22.517752 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:22.518569 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:22.519827 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:22.520384 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:22.520516 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:22.523784 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:22.523789 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:22.523792 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:22.523821 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:22.526641 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:22.527339 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:22.527351 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:22.527353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:22.527506 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:22.527914 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:22.527918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:22.527920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:22.528204 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:22.528386 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:22.528763 (+ 377us) server_negotiation.cc:299] Negotiation successful
0810 00:48:22.528987 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 00:48:22.529364 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:22.515831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:41333 (local address 127.0.0.1:49004)
0810 00:48:22.516013 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:22.516032 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:22.516135 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:22.516708 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:22.516712 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:22.517369 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:48:22.517610 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:22.517620 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:22.518703 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:22.518710 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:22.519676 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:22.519688 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:22.520539 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:22.520545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:22.520736 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:22.521350 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:48:22.521368 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:48:22.523596 (+ 2228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:48:22.526781 (+ 3185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:22.526788 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:22.526792 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:22.527214 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:22.527623 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:22.527628 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:22.527632 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:22.527792 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:22.528356 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:48:22.528361 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:48:22.528515 (+ 154us) client_negotiation.cc:769] Sending connection context
0810 00:48:22.528651 (+ 136us) client_negotiation.cc:240] Negotiation successful
0810 00:48:22.528968 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 00:48:22.535848 16283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea1e291e18b04f43af3b885eaead9c51: Bootstrap starting.
I20260810 00:48:22.539902 16283 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea1e291e18b04f43af3b885eaead9c51: Neither blocks nor log segments found. Creating new log.
I20260810 00:48:22.541190 16283 log.cc:824] T 00000000000000000000000000000000 P ea1e291e18b04f43af3b885eaead9c51: Log is configured to *not* fsync() on all Append() calls
I20260810 00:48:22.544725 16283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea1e291e18b04f43af3b885eaead9c51: No bootstrap required, opened a new log
I20260810 00:48:22.555783 16283 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea1e291e18b04f43af3b885eaead9c51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1e291e18b04f43af3b885eaead9c51" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34753 } } peers { permanent_uuid: "8904933c420149658a2a111ff8d2b6ae" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 40475 } } peers { permanent_uuid: "5d2e4f0de55f4f13a9e122ab0eb518dc" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 41333 } }
I20260810 00:48:22.556170 16283 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea1e291e18b04f43af3b885eaead9c51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:48:22.556298 16283 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea1e291e18b04f43af3b885eaead9c51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea1e291e18b04f43af3b885eaead9c51, State: Initialized, Role: FOLLOWER
I20260810 00:48:22.556927 16283 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea1e291e18b04f43af3b885eaead9c51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1e291e18b04f43af3b885eaead9c51" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34753 } } peers { permanent_uuid: "8904933c420149658a2a111ff8d2b6ae" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 40475 } } peers { permanent_uuid: "5d2e4f0de55f4f13a9e122ab0eb518dc" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 41333 } }
I20260810 00:48:22.558370 16550 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea1e291e18b04f43af3b885eaead9c51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1e291e18b04f43af3b885eaead9c51" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34753 } } peers { permanent_uuid: "8904933c420149658a2a111ff8d2b6ae" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 40475 } } peers { permanent_uuid: "5d2e4f0de55f4f13a9e122ab0eb518dc" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 41333 } } }
I20260810 00:48:22.558885 16550 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea1e291e18b04f43af3b885eaead9c51 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 00:48:22.559652 16283 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea1e291e18b04f43af3b885eaead9c51 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 00:48:22.572746 16576 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ea1e291e18b04f43af3b885eaead9c51: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 00:48:22.572965 16576 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 00:48:22.568614 16179 master_runner.cc:186] Error getting master registration for 127.28.6.61:40475: OK, instance_id { permanent_uuid: "8904933c420149658a2a111ff8d2b6ae" instance_seqno: 1786322901870414 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 00:48:22.581228 16179 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 00:48:22.593007 16395 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.60" port: 41333 } attempt: 5
W20260810 00:48:22.602065 16441 tablet.cc:2369] T 00000000000000000000000000000000 P 5d2e4f0de55f4f13a9e122ab0eb518dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 00:48:22 dist-test-slave-1nl6 krb5kdc[16159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322901, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.61@KRBTEST.COM for kudu/127.28.6.60@KRBTEST.COM
I20260810 00:48:22.610334 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:22.593576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:41333 (local address 127.0.0.1:49018)
0810 00:48:22.593834 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:22.593857 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:22.593981 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:22.594361 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:22.594367 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:22.594928 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:48:22.595150 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:22.595163 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:22.596626 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:22.596632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:22.598070 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:22.598086 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:22.599109 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:22.599115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:22.599329 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:22.600109 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:48:22.600135 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:48:22.604583 (+ 4448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:48:22.607888 (+ 3305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:22.607894 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:22.607898 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:22.608270 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:22.608654 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:22.608659 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:22.608663 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:22.608807 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:22.609328 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:48:22.609333 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:48:22.609479 (+ 146us) client_negotiation.cc:769] Sending connection context
0810 00:48:22.609644 (+ 165us) client_negotiation.cc:240] Negotiation successful
0810 00:48:22.609938 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260810 00:48:22.610358 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:22.593667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49018 (local address 127.28.6.60:41333)
0810 00:48:22.593854 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:22.593861 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:22.593924 (+ 63us) server_negotiation.cc:407] Connection header received
0810 00:48:22.594118 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:22.594124 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:22.594199 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:22.594311 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:22.595313 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:22.596463 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:22.598241 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:22.598943 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:22.599072 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:22.604780 (+ 5708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:22.604785 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:22.604788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:22.604818 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:22.607718 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:22.608397 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:22.608402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:22.608406 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:22.608529 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:22.608919 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:22.608924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:22.608928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:22.609199 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:22.609360 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:22.609756 (+ 396us) server_negotiation.cc:299] Negotiation successful
0810 00:48:22.609989 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 00:48:22.613907 16532 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d2e4f0de55f4f13a9e122ab0eb518dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:48:22.614180 16532 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d2e4f0de55f4f13a9e122ab0eb518dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1e291e18b04f43af3b885eaead9c51" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34753 } } peers { permanent_uuid: "8904933c420149658a2a111ff8d2b6ae" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 40475 } } peers { permanent_uuid: "5d2e4f0de55f4f13a9e122ab0eb518dc" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 41333 } }
I20260810 00:48:22.615865 16395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8904933c420149658a2a111ff8d2b6ae: Bootstrap starting.
I20260810 00:48:22.615870 16532 leader_election.cc:288] T 00000000000000000000000000000000 P 5d2e4f0de55f4f13a9e122ab0eb518dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea1e291e18b04f43af3b885eaead9c51 (127.28.6.62:34753), 8904933c420149658a2a111ff8d2b6ae (127.28.6.61:40475)
I20260810 00:48:22.617759 16246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5d2e4f0de55f4f13a9e122ab0eb518dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea1e291e18b04f43af3b885eaead9c51" is_pre_election: true
I20260810 00:48:22.618415 16246 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ea1e291e18b04f43af3b885eaead9c51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d2e4f0de55f4f13a9e122ab0eb518dc in term 0.
I20260810 00:48:22.619168 16438 leader_election.cc:302] T 00000000000000000000000000000000 P 5d2e4f0de55f4f13a9e122ab0eb518dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d2e4f0de55f4f13a9e122ab0eb518dc, ea1e291e18b04f43af3b885eaead9c51; no voters:
I20260810 00:48:22.619659 16532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d2e4f0de55f4f13a9e122ab0eb518dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 00:48:22.619825 16532 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d2e4f0de55f4f13a9e122ab0eb518dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 00:48:22.619947 16532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d2e4f0de55f4f13a9e122ab0eb518dc [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:48:22.620178 16395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8904933c420149658a2a111ff8d2b6ae: Neither blocks nor log segments found. Creating new log.
I20260810 00:48:22.621479 16395 log.cc:824] T 00000000000000000000000000000000 P 8904933c420149658a2a111ff8d2b6ae: Log is configured to *not* fsync() on all Append() calls
W20260810 00:48:22.622182 16211 tablet.cc:2369] T 00000000000000000000000000000000 P ea1e291e18b04f43af3b885eaead9c51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:48:22.628571 16532 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d2e4f0de55f4f13a9e122ab0eb518dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1e291e18b04f43af3b885eaead9c51" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34753 } } peers { permanent_uuid: "8904933c420149658a2a111ff8d2b6ae" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 40475 } } peers { permanent_uuid: "5d2e4f0de55f4f13a9e122ab0eb518dc" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 41333 } }
I20260810 00:48:22.630949 16246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5d2e4f0de55f4f13a9e122ab0eb518dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea1e291e18b04f43af3b885eaead9c51"
I20260810 00:48:22.631213 16246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea1e291e18b04f43af3b885eaead9c51 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:48:22.636811 16246 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ea1e291e18b04f43af3b885eaead9c51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d2e4f0de55f4f13a9e122ab0eb518dc in term 1.
I20260810 00:48:22.637709 16438 leader_election.cc:302] T 00000000000000000000000000000000 P 5d2e4f0de55f4f13a9e122ab0eb518dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d2e4f0de55f4f13a9e122ab0eb518dc, ea1e291e18b04f43af3b885eaead9c51; no voters:
I20260810 00:48:22.642997 16579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d2e4f0de55f4f13a9e122ab0eb518dc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:48:22.648054 16532 leader_election.cc:288] T 00000000000000000000000000000000 P 5d2e4f0de55f4f13a9e122ab0eb518dc [CANDIDATE]: Term 1 election: Requested vote from peers ea1e291e18b04f43af3b885eaead9c51 (127.28.6.62:34753), 8904933c420149658a2a111ff8d2b6ae (127.28.6.61:40475)
I20260810 00:48:22.648294 16395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8904933c420149658a2a111ff8d2b6ae: No bootstrap required, opened a new log
I20260810 00:48:22.659201 16579 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d2e4f0de55f4f13a9e122ab0eb518dc [term 1 LEADER]: Becoming Leader. State: Replica: 5d2e4f0de55f4f13a9e122ab0eb518dc, State: Running, Role: LEADER
I20260810 00:48:22.659847 16579 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d2e4f0de55f4f13a9e122ab0eb518dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1e291e18b04f43af3b885eaead9c51" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34753 } } peers { permanent_uuid: "8904933c420149658a2a111ff8d2b6ae" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 40475 } } peers { permanent_uuid: "5d2e4f0de55f4f13a9e122ab0eb518dc" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 41333 } }
I20260810 00:48:22.663905 16395 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8904933c420149658a2a111ff8d2b6ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1e291e18b04f43af3b885eaead9c51" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34753 } } peers { permanent_uuid: "8904933c420149658a2a111ff8d2b6ae" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 40475 } } peers { permanent_uuid: "5d2e4f0de55f4f13a9e122ab0eb518dc" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 41333 } }
I20260810 00:48:22.664345 16395 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8904933c420149658a2a111ff8d2b6ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:48:22.664464 16395 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8904933c420149658a2a111ff8d2b6ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8904933c420149658a2a111ff8d2b6ae, State: Initialized, Role: FOLLOWER
I20260810 00:48:22.665194 16395 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8904933c420149658a2a111ff8d2b6ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1e291e18b04f43af3b885eaead9c51" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34753 } } peers { permanent_uuid: "8904933c420149658a2a111ff8d2b6ae" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 40475 } } peers { permanent_uuid: "5d2e4f0de55f4f13a9e122ab0eb518dc" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 41333 } }
I20260810 00:48:22.674456 16532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d2e4f0de55f4f13a9e122ab0eb518dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d2e4f0de55f4f13a9e122ab0eb518dc. Latest consensus state: current_term: 1 leader_uuid: "5d2e4f0de55f4f13a9e122ab0eb518dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1e291e18b04f43af3b885eaead9c51" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34753 } } peers { permanent_uuid: "8904933c420149658a2a111ff8d2b6ae" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 40475 } } peers { permanent_uuid: "5d2e4f0de55f4f13a9e122ab0eb518dc" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 41333 } } }
I20260810 00:48:22.674774 16532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d2e4f0de55f4f13a9e122ab0eb518dc [sys.catalog]: This master's current role is: LEADER
I20260810 00:48:22.687487 16583 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:48:22.687410 16582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8904933c420149658a2a111ff8d2b6ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1e291e18b04f43af3b885eaead9c51" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34753 } } peers { permanent_uuid: "8904933c420149658a2a111ff8d2b6ae" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 40475 } } peers { permanent_uuid: "5d2e4f0de55f4f13a9e122ab0eb518dc" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 41333 } } }
I20260810 00:48:22.688026 16582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8904933c420149658a2a111ff8d2b6ae [sys.catalog]: This master's current role is: FOLLOWER
I20260810 00:48:22.688972 16395 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8904933c420149658a2a111ff8d2b6ae [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:48:22.689007 16357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5d2e4f0de55f4f13a9e122ab0eb518dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8904933c420149658a2a111ff8d2b6ae"
I20260810 00:48:22.689038 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5d2e4f0de55f4f13a9e122ab0eb518dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8904933c420149658a2a111ff8d2b6ae" is_pre_election: true
I20260810 00:48:22.689554 16357 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8904933c420149658a2a111ff8d2b6ae [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:48:22.693799 16357 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8904933c420149658a2a111ff8d2b6ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d2e4f0de55f4f13a9e122ab0eb518dc in term 1.
I20260810 00:48:22.696372 16583 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:48:22.711773 16357 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8904933c420149658a2a111ff8d2b6ae [term 1 FOLLOWER]: Refusing update from remote peer 5d2e4f0de55f4f13a9e122ab0eb518dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:48:22.712745 16532 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5d2e4f0de55f4f13a9e122ab0eb518dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8904933c420149658a2a111ff8d2b6ae" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 40475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:48:22.724006 16246 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ea1e291e18b04f43af3b885eaead9c51 [term 1 FOLLOWER]: Refusing update from remote peer 5d2e4f0de55f4f13a9e122ab0eb518dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:48:22.725006 16532 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5d2e4f0de55f4f13a9e122ab0eb518dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea1e291e18b04f43af3b885eaead9c51" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:48:22.737915 16550 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea1e291e18b04f43af3b885eaead9c51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d2e4f0de55f4f13a9e122ab0eb518dc. Latest consensus state: current_term: 1 leader_uuid: "5d2e4f0de55f4f13a9e122ab0eb518dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1e291e18b04f43af3b885eaead9c51" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34753 } } peers { permanent_uuid: "8904933c420149658a2a111ff8d2b6ae" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 40475 } } peers { permanent_uuid: "5d2e4f0de55f4f13a9e122ab0eb518dc" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 41333 } } }
I20260810 00:48:22.738253 16550 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea1e291e18b04f43af3b885eaead9c51 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 00:48:22.743407 16582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8904933c420149658a2a111ff8d2b6ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d2e4f0de55f4f13a9e122ab0eb518dc. Latest consensus state: current_term: 1 leader_uuid: "5d2e4f0de55f4f13a9e122ab0eb518dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1e291e18b04f43af3b885eaead9c51" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34753 } } peers { permanent_uuid: "8904933c420149658a2a111ff8d2b6ae" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 40475 } } peers { permanent_uuid: "5d2e4f0de55f4f13a9e122ab0eb518dc" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 41333 } } }
I20260810 00:48:22.743790 16582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8904933c420149658a2a111ff8d2b6ae [sys.catalog]: This master's current role is: FOLLOWER
I20260810 00:48:22.745612 16532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d2e4f0de55f4f13a9e122ab0eb518dc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5d2e4f0de55f4f13a9e122ab0eb518dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1e291e18b04f43af3b885eaead9c51" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34753 } } peers { permanent_uuid: "8904933c420149658a2a111ff8d2b6ae" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 40475 } } peers { permanent_uuid: "5d2e4f0de55f4f13a9e122ab0eb518dc" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 41333 } } }
I20260810 00:48:22.745971 16532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d2e4f0de55f4f13a9e122ab0eb518dc [sys.catalog]: This master's current role is: LEADER
I20260810 00:48:22.746261 16532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d2e4f0de55f4f13a9e122ab0eb518dc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5d2e4f0de55f4f13a9e122ab0eb518dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1e291e18b04f43af3b885eaead9c51" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34753 } } peers { permanent_uuid: "8904933c420149658a2a111ff8d2b6ae" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 40475 } } peers { permanent_uuid: "5d2e4f0de55f4f13a9e122ab0eb518dc" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 41333 } } }
I20260810 00:48:22.746563 16532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d2e4f0de55f4f13a9e122ab0eb518dc [sys.catalog]: This master's current role is: LEADER
I20260810 00:48:22.748383 16583 catalog_manager.cc:1347] Generated new cluster ID: aec9bfbccdc3454ba04a6b875076dee8
I20260810 00:48:22.748503 16583 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:48:22.772145 16550 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea1e291e18b04f43af3b885eaead9c51 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5d2e4f0de55f4f13a9e122ab0eb518dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1e291e18b04f43af3b885eaead9c51" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34753 } } peers { permanent_uuid: "8904933c420149658a2a111ff8d2b6ae" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 40475 } } peers { permanent_uuid: "5d2e4f0de55f4f13a9e122ab0eb518dc" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 41333 } } }
I20260810 00:48:22.772475 16550 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea1e291e18b04f43af3b885eaead9c51 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 00:48:22.782001 16582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8904933c420149658a2a111ff8d2b6ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5d2e4f0de55f4f13a9e122ab0eb518dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1e291e18b04f43af3b885eaead9c51" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34753 } } peers { permanent_uuid: "8904933c420149658a2a111ff8d2b6ae" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 40475 } } peers { permanent_uuid: "5d2e4f0de55f4f13a9e122ab0eb518dc" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 41333 } } }
I20260810 00:48:22.782274 16582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8904933c420149658a2a111ff8d2b6ae [sys.catalog]: This master's current role is: FOLLOWER
I20260810 00:48:22.809172 16583 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:48:22.812335 16583 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:48:22.816466 16613 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 00:48:22.841600 16583 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d2e4f0de55f4f13a9e122ab0eb518dc: Generated new TSK 0
I20260810 00:48:22.842832 16583 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 00:48:23.575235 16576 catalog_manager.cc:1259] Loaded cluster ID: aec9bfbccdc3454ba04a6b875076dee8
I20260810 00:48:23.575376 16576 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ea1e291e18b04f43af3b885eaead9c51: loading cluster ID for follower catalog manager: success
I20260810 00:48:23.578285 16576 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ea1e291e18b04f43af3b885eaead9c51: acquiring CA information for follower catalog manager: success
I20260810 00:48:23.580102 16576 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ea1e291e18b04f43af3b885eaead9c51: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 00:48:23.821069 16613 catalog_manager.cc:1259] Loaded cluster ID: aec9bfbccdc3454ba04a6b875076dee8
I20260810 00:48:23.821208 16613 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8904933c420149658a2a111ff8d2b6ae: loading cluster ID for follower catalog manager: success
I20260810 00:48:23.823876 16613 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8904933c420149658a2a111ff8d2b6ae: acquiring CA information for follower catalog manager: success
I20260810 00:48:23.825732 16613 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8904933c420149658a2a111ff8d2b6ae: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 00:48:23.956189 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 16179
I20260810 00:48:23.957463 16275 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:48:24.068351 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 16179
I20260810 00:48:24.100517 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 16291
I20260810 00:48:24.101776 16389 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:48:24.224534 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 16291
I20260810 00:48:24.251077 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 16407
I20260810 00:48:24.252223 16507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:48:24.361882 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 16407
2026-08-10T00:48:24Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4958 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:48:24 dist-test-slave-1nl6 krb5kdc[16635](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:48:24 dist-test-slave-1nl6 krb5kdc[16635](info): set up 2 sockets
Aug 10 00:48:24 dist-test-slave-1nl6 krb5kdc[16635](info): commencing operation
krb5kdc: starting...
W20260810 00:48:25.661609 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.253s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:48:25 dist-test-slave-1nl6 krb5kdc[16635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:48:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:48:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.6.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
I20260810 00:48:25.923662 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:43527
--webserver_interface=127.28.6.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:40141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:43527
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:48:26.267736 16655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:48:26.268002 16655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:48:26.268091 16655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:48:26.275034 16655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:48:26.275130 16655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:48:26.275178 16655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:48:26.275242 16655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:48:26.275287 16655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:48:26.284559 16655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:40141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:43527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:43527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:48:26.285770 16655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:48:26.287101 16655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:48:26.298796 16662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:26.298884 16663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:26.298926 16665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:48:26.299715 16655 server_base.cc:1034] running on GCE node
I20260810 00:48:26.300474 16655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:48:26.301803 16655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:48:26.303014 16655 hybrid_clock.cc:648] HybridClock initialized: now 1786322906302948 us; error 45 us; skew 500 ppm
I20260810 00:48:26.303366 16655 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:48:26 dist-test-slave-1nl6 krb5kdc[16635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322906, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:48:26.306443 16655 init.cc:377] Logged in from keytab as kudu/127.28.6.62@KRBTEST.COM (short username kudu)
I20260810 00:48:26.307962 16655 webserver.cc:460] Webserver started at http://127.28.6.62:39093/ using document root <none> and password file <none>
I20260810 00:48:26.308627 16655 fs_manager.cc:359] Metadata directory not provided
I20260810 00:48:26.308801 16655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:48:26.309067 16655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:48:26.311991 16655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "164e2b7cd6f542878c1f0dfab21a6ca4"
format_stamp: "Formatted at 2026-08-10 00:48:26 on dist-test-slave-1nl6"
server_key: "d03d79100418d3daff7647210567ad8b"
server_key_iv: "d610df5f201d4e4541892dddb2a72646"
server_key_version: "encryptionkey@0"
I20260810 00:48:26.312702 16655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "164e2b7cd6f542878c1f0dfab21a6ca4"
format_stamp: "Formatted at 2026-08-10 00:48:26 on dist-test-slave-1nl6"
server_key: "d03d79100418d3daff7647210567ad8b"
server_key_iv: "d610df5f201d4e4541892dddb2a72646"
server_key_version: "encryptionkey@0"
I20260810 00:48:26.317896 16655 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 00:48:26.321269 16675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:26.322504 16655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260810 00:48:26.322700 16655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "164e2b7cd6f542878c1f0dfab21a6ca4"
format_stamp: "Formatted at 2026-08-10 00:48:26 on dist-test-slave-1nl6"
server_key: "d03d79100418d3daff7647210567ad8b"
server_key_iv: "d610df5f201d4e4541892dddb2a72646"
server_key_version: "encryptionkey@0"
I20260810 00:48:26.322907 16655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:48:26.341171 16655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:48:26.344937 16655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:48:26.345232 16655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:48:26.363819 16655 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:43527
I20260810 00:48:26.363855 16755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:43527 every 8 connection(s)
I20260810 00:48:26.365392 16655 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:48:26.367502 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 16655
I20260810 00:48:26.367745 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:48:26.368449 28696 external_mini_cluster.cc:1428] Setting key fa17533a2e32f9f0d55c6d0b2f4d87a1
I20260810 00:48:26.369632 16756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 00:48:26 dist-test-slave-1nl6 krb5kdc[16635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:48:26.383587 16756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 164e2b7cd6f542878c1f0dfab21a6ca4: Bootstrap starting.
I20260810 00:48:26.387483 16756 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 164e2b7cd6f542878c1f0dfab21a6ca4: Neither blocks nor log segments found. Creating new log.
I20260810 00:48:26.388794 16756 log.cc:824] T 00000000000000000000000000000000 P 164e2b7cd6f542878c1f0dfab21a6ca4: Log is configured to *not* fsync() on all Append() calls
I20260810 00:48:26.392141 16756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 164e2b7cd6f542878c1f0dfab21a6ca4: No bootstrap required, opened a new log
I20260810 00:48:26.393231 16759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:26.370423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33578 (local address 127.28.6.62:43527)
0810 00:48:26.371296 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:26.371316 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:26.371370 (+ 54us) server_negotiation.cc:407] Connection header received
0810 00:48:26.373231 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:26.373260 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:26.373761 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:26.374201 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:26.375655 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:26.377657 (+ 2002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:26.378996 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:26.380061 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:26.380303 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:26.384548 (+ 4245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:26.384559 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:26.384576 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:26.384621 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:26.388583 (+ 3962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:26.389425 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:26.389433 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:26.389447 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:26.389621 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:26.390100 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:26.390105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:26.390108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:26.390772 (+ 664us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:26.391109 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:26.391493 (+ 384us) server_negotiation.cc:299] Negotiation successful
0810 00:48:26.391910 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":634,"thread_start_us":399,"threads_started":1}
I20260810 00:48:26.397014 28696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 00:48:26.400592 16756 raft_consensus.cc:348] T 00000000000000000000000000000000 P 164e2b7cd6f542878c1f0dfab21a6ca4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164e2b7cd6f542878c1f0dfab21a6ca4" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 43527 } }
I20260810 00:48:26.400928 16756 raft_consensus.cc:374] T 00000000000000000000000000000000 P 164e2b7cd6f542878c1f0dfab21a6ca4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:48:26.401042 16756 raft_consensus.cc:729] T 00000000000000000000000000000000 P 164e2b7cd6f542878c1f0dfab21a6ca4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 164e2b7cd6f542878c1f0dfab21a6ca4, State: Initialized, Role: FOLLOWER
I20260810 00:48:26.401558 16756 consensus_queue.cc:260] T 00000000000000000000000000000000 P 164e2b7cd6f542878c1f0dfab21a6ca4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164e2b7cd6f542878c1f0dfab21a6ca4" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 43527 } }
I20260810 00:48:26.401779 16756 raft_consensus.cc:388] T 00000000000000000000000000000000 P 164e2b7cd6f542878c1f0dfab21a6ca4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:48:26.401933 16756 raft_consensus.cc:482] T 00000000000000000000000000000000 P 164e2b7cd6f542878c1f0dfab21a6ca4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:48:26.402093 16756 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 164e2b7cd6f542878c1f0dfab21a6ca4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:48:26.404420 16756 raft_consensus.cc:504] T 00000000000000000000000000000000 P 164e2b7cd6f542878c1f0dfab21a6ca4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164e2b7cd6f542878c1f0dfab21a6ca4" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 43527 } }
I20260810 00:48:26.405014 16756 leader_election.cc:302] T 00000000000000000000000000000000 P 164e2b7cd6f542878c1f0dfab21a6ca4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 164e2b7cd6f542878c1f0dfab21a6ca4; no voters:
I20260810 00:48:26.405652 16756 leader_election.cc:288] T 00000000000000000000000000000000 P 164e2b7cd6f542878c1f0dfab21a6ca4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:48:26.405755 16761 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 164e2b7cd6f542878c1f0dfab21a6ca4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:48:26.406484 16761 raft_consensus.cc:686] T 00000000000000000000000000000000 P 164e2b7cd6f542878c1f0dfab21a6ca4 [term 1 LEADER]: Becoming Leader. State: Replica: 164e2b7cd6f542878c1f0dfab21a6ca4, State: Running, Role: LEADER
I20260810 00:48:26.407114 16761 consensus_queue.cc:237] T 00000000000000000000000000000000 P 164e2b7cd6f542878c1f0dfab21a6ca4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164e2b7cd6f542878c1f0dfab21a6ca4" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 43527 } }
I20260810 00:48:26.407769 16756 sys_catalog.cc:564] T 00000000000000000000000000000000 P 164e2b7cd6f542878c1f0dfab21a6ca4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:48:26.410283 16763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 164e2b7cd6f542878c1f0dfab21a6ca4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 164e2b7cd6f542878c1f0dfab21a6ca4. Latest consensus state: current_term: 1 leader_uuid: "164e2b7cd6f542878c1f0dfab21a6ca4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164e2b7cd6f542878c1f0dfab21a6ca4" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 43527 } } }
I20260810 00:48:26.410250 16762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 164e2b7cd6f542878c1f0dfab21a6ca4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "164e2b7cd6f542878c1f0dfab21a6ca4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164e2b7cd6f542878c1f0dfab21a6ca4" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 43527 } } }
I20260810 00:48:26.410676 16763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 164e2b7cd6f542878c1f0dfab21a6ca4 [sys.catalog]: This master's current role is: LEADER
I20260810 00:48:26.410722 16762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 164e2b7cd6f542878c1f0dfab21a6ca4 [sys.catalog]: This master's current role is: LEADER
I20260810 00:48:26.416827 16769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:48:26.422317 16769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:48:26.430691 16769 catalog_manager.cc:1347] Generated new cluster ID: 0f96958127f54539aec200a12b7fe8fe
I20260810 00:48:26.430795 16769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:48:26.441792 16769 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:48:26.443645 16769 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:48:26.459632 16769 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 164e2b7cd6f542878c1f0dfab21a6ca4: Generated new TSK 0
I20260810 00:48:26.460634 16769 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 00:48:26.509320 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 16655
I20260810 00:48:26.510560 16748 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:48:26.604897 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 16655
2026-08-10T00:48:26Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2243 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:48:26 dist-test-slave-1nl6 krb5kdc[16802](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:48:26 dist-test-slave-1nl6 krb5kdc[16802](info): set up 2 sockets
Aug 10 00:48:26 dist-test-slave-1nl6 krb5kdc[16802](info): commencing operation
krb5kdc: starting...
W20260810 00:48:27.926319 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.274s user 0.004s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:48:28 dist-test-slave-1nl6 krb5kdc[16802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:48:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:48:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.6.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
I20260810 00:48:28.194386 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:36385
--webserver_interface=127.28.6.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:44267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:36385
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:48:28.526240 16821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:48:28.526566 16821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:48:28.526726 16821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:48:28.533843 16821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:48:28.533941 16821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:48:28.534081 16821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:48:28.534176 16821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:48:28.534245 16821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:48:28.543613 16821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:44267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:36385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:36385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:48:28.544893 16821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:48:28.546250 16821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:48:28.557662 16831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:28.557942 16827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:28.557942 16828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:48:28.558887 16821 server_base.cc:1034] running on GCE node
I20260810 00:48:28.559746 16821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:48:28.561054 16821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:48:28.562291 16821 hybrid_clock.cc:648] HybridClock initialized: now 1786322908562245 us; error 47 us; skew 500 ppm
I20260810 00:48:28.562647 16821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:48:28 dist-test-slave-1nl6 krb5kdc[16802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322908, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:48:28.565685 16821 init.cc:377] Logged in from keytab as kudu/127.28.6.62@KRBTEST.COM (short username kudu)
I20260810 00:48:28.567233 16821 webserver.cc:460] Webserver started at http://127.28.6.62:43747/ using document root <none> and password file <none>
I20260810 00:48:28.567891 16821 fs_manager.cc:359] Metadata directory not provided
I20260810 00:48:28.568038 16821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:48:28.568352 16821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:48:28.571282 16821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "4fd535fd266f44949f3ea71c1aefee5d"
format_stamp: "Formatted at 2026-08-10 00:48:28 on dist-test-slave-1nl6"
server_key: "2171d19594674b98514a47a2cade09e3"
server_key_iv: "50b9e3e15cbf1da654407d5cac7b55d8"
server_key_version: "encryptionkey@0"
I20260810 00:48:28.572048 16821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "4fd535fd266f44949f3ea71c1aefee5d"
format_stamp: "Formatted at 2026-08-10 00:48:28 on dist-test-slave-1nl6"
server_key: "2171d19594674b98514a47a2cade09e3"
server_key_iv: "50b9e3e15cbf1da654407d5cac7b55d8"
server_key_version: "encryptionkey@0"
I20260810 00:48:28.577190 16821 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 00:48:28.580444 16841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:28.581634 16821 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 00:48:28.581810 16821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "4fd535fd266f44949f3ea71c1aefee5d"
format_stamp: "Formatted at 2026-08-10 00:48:28 on dist-test-slave-1nl6"
server_key: "2171d19594674b98514a47a2cade09e3"
server_key_iv: "50b9e3e15cbf1da654407d5cac7b55d8"
server_key_version: "encryptionkey@0"
I20260810 00:48:28.581997 16821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:48:28.601716 16821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:48:28.605476 16821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:48:28.605769 16821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:48:28.624112 16821 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:36385
I20260810 00:48:28.624138 16919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:36385 every 8 connection(s)
I20260810 00:48:28.625418 16821 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:48:28.628546 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 16821
I20260810 00:48:28.628803 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:48:28.629480 16920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:48:28.629680 28696 external_mini_cluster.cc:1428] Setting key 0b5bfbbfbe4d61b27b606d88e0f423c9
I20260810 00:48:28.641803 16920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d: Bootstrap starting.
Aug 10 00:48:28 dist-test-slave-1nl6 krb5kdc[16802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:48:28.645351 16920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d: Neither blocks nor log segments found. Creating new log.
I20260810 00:48:28.646528 16920 log.cc:824] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d: Log is configured to *not* fsync() on all Append() calls
I20260810 00:48:28.649773 16920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d: No bootstrap required, opened a new log
I20260810 00:48:28.652386 16925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:28.631778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54136 (local address 127.28.6.62:36385)
0810 00:48:28.632536 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:28.632547 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:28.632589 (+ 42us) server_negotiation.cc:407] Connection header received
0810 00:48:28.634346 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:28.634367 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:28.634706 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:28.635169 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:28.636664 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:28.637940 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:28.639339 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:28.640061 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:28.640260 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:28.644699 (+ 4439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:28.644707 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:28.644720 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:28.644757 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:28.648178 (+ 3421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:28.649057 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:28.649063 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:28.649069 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:28.649225 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:28.649658 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:28.649661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:28.649663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:28.650204 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:28.650431 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:28.651009 (+ 578us) server_negotiation.cc:299] Negotiation successful
0810 00:48:28.651405 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":350,"threads_started":1}
I20260810 00:48:28.658420 16920 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd535fd266f44949f3ea71c1aefee5d" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36385 } }
I20260810 00:48:28.658783 16920 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:48:28.658929 16920 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd535fd266f44949f3ea71c1aefee5d, State: Initialized, Role: FOLLOWER
I20260810 00:48:28.659469 16920 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd535fd266f44949f3ea71c1aefee5d" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36385 } }
I20260810 00:48:28.659711 16920 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:48:28.659860 16920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:48:28.660041 16920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:48:28.662309 16920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd535fd266f44949f3ea71c1aefee5d" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36385 } }
I20260810 00:48:28.662813 16920 leader_election.cc:302] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fd535fd266f44949f3ea71c1aefee5d; no voters:
I20260810 00:48:28.663417 16920 leader_election.cc:288] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:48:28.663538 16928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:48:28.664336 16928 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [term 1 LEADER]: Becoming Leader. State: Replica: 4fd535fd266f44949f3ea71c1aefee5d, State: Running, Role: LEADER
I20260810 00:48:28.664978 16928 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd535fd266f44949f3ea71c1aefee5d" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36385 } }
I20260810 00:48:28.665627 16920 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:48:28.667641 16933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fd535fd266f44949f3ea71c1aefee5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd535fd266f44949f3ea71c1aefee5d" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36385 } } }
I20260810 00:48:28.667964 16933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [sys.catalog]: This master's current role is: LEADER
I20260810 00:48:28.669876 16934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fd535fd266f44949f3ea71c1aefee5d. Latest consensus state: current_term: 1 leader_uuid: "4fd535fd266f44949f3ea71c1aefee5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd535fd266f44949f3ea71c1aefee5d" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36385 } } }
I20260810 00:48:28.670212 16944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:48:28.670218 16934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [sys.catalog]: This master's current role is: LEADER
I20260810 00:48:28.675645 16944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:48:28.685022 16944 catalog_manager.cc:1347] Generated new cluster ID: 5bee55c23a1e4e328058c3e0186cb203
I20260810 00:48:28.685124 16944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:48:28.695039 16944 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:48:28.696192 16944 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:48:28.705662 16944 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d: Generated new TSK 0
I20260810 00:48:28.706621 16944 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.6.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab.
Entry for principal kudu/127.28.6.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab.
I20260810 00:48:28.774644 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.1:0
--local_ip_for_outbound_sockets=127.28.6.1
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:36385
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:44267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:48:29.111303 16969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:48:29.111567 16969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:48:29.111737 16969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:48:29.118813 16969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:48:29.118927 16969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:48:29.119099 16969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.1
I20260810 00:48:29.128563 16969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:44267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:36385
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.1
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:48:29.129776 16969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:48:29.131136 16969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:48:29.143956 16976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:29.144024 16980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:29.143989 16977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:48:29.144986 16969 server_base.cc:1034] running on GCE node
I20260810 00:48:29.145614 16969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:48:29.147189 16969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:48:29.148478 16969 hybrid_clock.cc:648] HybridClock initialized: now 1786322909148352 us; error 90 us; skew 500 ppm
I20260810 00:48:29.148857 16969 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:48:29 dist-test-slave-1nl6 krb5kdc[16802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322909, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:48:29.152253 16969 init.cc:377] Logged in from keytab as kudu/127.28.6.1@KRBTEST.COM (short username kudu)
I20260810 00:48:29.153961 16969 webserver.cc:460] Webserver started at http://127.28.6.1:42113/ using document root <none> and password file <none>
I20260810 00:48:29.154634 16969 fs_manager.cc:359] Metadata directory not provided
I20260810 00:48:29.154795 16969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:48:29.155139 16969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:48:29.157938 16969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/data/instance:
uuid: "5baea45d390a4f0ab315bcf66cf1687a"
format_stamp: "Formatted at 2026-08-10 00:48:29 on dist-test-slave-1nl6"
server_key: "1ef7b79bab9d263431028abc0ea015d8"
server_key_iv: "1077defa0428963b27f8169416d1e5df"
server_key_version: "encryptionkey@0"
I20260810 00:48:29.158737 16969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance:
uuid: "5baea45d390a4f0ab315bcf66cf1687a"
format_stamp: "Formatted at 2026-08-10 00:48:29 on dist-test-slave-1nl6"
server_key: "1ef7b79bab9d263431028abc0ea015d8"
server_key_iv: "1077defa0428963b27f8169416d1e5df"
server_key_version: "encryptionkey@0"
I20260810 00:48:29.163744 16969 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260810 00:48:29.167299 16988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:29.168697 16969 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 00:48:29.168866 16969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/wal
uuid: "5baea45d390a4f0ab315bcf66cf1687a"
format_stamp: "Formatted at 2026-08-10 00:48:29 on dist-test-slave-1nl6"
server_key: "1ef7b79bab9d263431028abc0ea015d8"
server_key_iv: "1077defa0428963b27f8169416d1e5df"
server_key_version: "encryptionkey@0"
I20260810 00:48:29.169067 16969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:48:29.191721 16969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:48:29.196069 16969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:48:29.196364 16969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:48:29.197280 16969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:48:29.198562 16969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:48:29.198633 16969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:29.198800 16969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:48:29.198863 16969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:29.230799 16969 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.1:39615
I20260810 00:48:29.230881 17165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.1:39615 every 8 connection(s)
I20260810 00:48:29.232071 16969 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
I20260810 00:48:29.240263 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 16969
I20260810 00:48:29.240495 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance
I20260810 00:48:29.241149 28696 external_mini_cluster.cc:1428] Setting key 34dd9db181b70c1e1b28a096248a3ff2
Aug 10 00:48:29 dist-test-slave-1nl6 krb5kdc[16802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322909, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:48:29.252846 17168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:29.234542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:52045 (local address 127.28.6.62:36385)
0810 00:48:29.235013 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:29.235017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:29.235925 (+ 908us) server_negotiation.cc:407] Connection header received
0810 00:48:29.237688 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:29.237694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:29.237783 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:29.237903 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:29.239443 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:29.240153 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:29.241825 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:29.242242 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:29.242401 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:29.245743 (+ 3342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:29.245750 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:29.245756 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:29.245798 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:29.249414 (+ 3616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:29.250093 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:29.250101 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:29.250108 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:29.250268 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:29.250739 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:29.250747 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:29.250752 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:29.251163 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:29.251336 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:29.252106 (+ 770us) server_negotiation.cc:299] Negotiation successful
0810 00:48:29.252364 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":299,"thread_start_us":199,"threads_started":1}
I20260810 00:48:29.253644 17169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:29.235092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.1:52045)
0810 00:48:29.235772 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:29.235814 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:29.237429 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:29.238140 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:29.238148 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:29.238659 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:48:29.239279 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:29.239296 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:29.240332 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:29.240339 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:29.241656 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:29.241668 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:29.242477 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:29.242484 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:29.242721 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:29.243470 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:48:29.243497 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:48:29.245542 (+ 2045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:48:29.249546 (+ 4004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:29.249552 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:29.249562 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:29.249959 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:29.250390 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:29.250397 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:29.250403 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:29.250615 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:29.251290 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:48:29.251300 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:48:29.251683 (+ 383us) client_negotiation.cc:769] Sending connection context
0810 00:48:29.251947 (+ 264us) client_negotiation.cc:240] Negotiation successful
0810 00:48:29.252444 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":263,"threads_started":1}
I20260810 00:48:29.255146 17166 heartbeater.cc:347] Connected to a master server at 127.28.6.62:36385
I20260810 00:48:29.255455 17166 heartbeater.cc:464] Registering TS with master...
I20260810 00:48:29.256179 17166 heartbeater.cc:511] Master 127.28.6.62:36385 requested a full tablet report, sending...
I20260810 00:48:29.258239 16863 ts_manager.cc:194] Registered new tserver with Master: 5baea45d390a4f0ab315bcf66cf1687a (127.28.6.1:39615)
I20260810 00:48:29.259712 16863 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.1@KRBTEST.COM'} at 127.28.6.1:52045
WARNING: no policy specified for kudu/127.28.6.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab.
Entry for principal kudu/127.28.6.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab.
I20260810 00:48:29.299309 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.2:0
--local_ip_for_outbound_sockets=127.28.6.2
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:36385
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:44267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:48:29.645370 17172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:48:29.645689 17172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:48:29.645861 17172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:48:29.653146 17172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:48:29.653303 17172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:48:29.653479 17172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.2
I20260810 00:48:29.662930 17172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:44267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:36385
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.2
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:48:29.664178 17172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:48:29.665441 17172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:48:29.678972 17182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:29.678920 17188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:29.679111 17184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:48:29.679881 17172 server_base.cc:1034] running on GCE node
I20260810 00:48:29.680451 17172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:48:29.681414 17172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:48:29.682643 17172 hybrid_clock.cc:648] HybridClock initialized: now 1786322909682578 us; error 56 us; skew 500 ppm
I20260810 00:48:29.683020 17172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:48:29 dist-test-slave-1nl6 krb5kdc[16802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322909, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:48:29.689764 17172 init.cc:377] Logged in from keytab as kudu/127.28.6.2@KRBTEST.COM (short username kudu)
I20260810 00:48:29.691246 17172 webserver.cc:460] Webserver started at http://127.28.6.2:40167/ using document root <none> and password file <none>
I20260810 00:48:29.691866 17172 fs_manager.cc:359] Metadata directory not provided
I20260810 00:48:29.691970 17172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:48:29.692230 17172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:48:29.695118 17172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/data/instance:
uuid: "d3b6361ac1dd42839442096b8b99e033"
format_stamp: "Formatted at 2026-08-10 00:48:29 on dist-test-slave-1nl6"
server_key: "086f064ec1d4b39c7d60f3b44315d9b4"
server_key_iv: "259a9c38e028019e411488e8107ac4a4"
server_key_version: "encryptionkey@0"
I20260810 00:48:29.695868 17172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance:
uuid: "d3b6361ac1dd42839442096b8b99e033"
format_stamp: "Formatted at 2026-08-10 00:48:29 on dist-test-slave-1nl6"
server_key: "086f064ec1d4b39c7d60f3b44315d9b4"
server_key_iv: "259a9c38e028019e411488e8107ac4a4"
server_key_version: "encryptionkey@0"
I20260810 00:48:29.700739 17172 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 00:48:29.704025 17198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:29.705313 17172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 00:48:29.705497 17172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/wal
uuid: "d3b6361ac1dd42839442096b8b99e033"
format_stamp: "Formatted at 2026-08-10 00:48:29 on dist-test-slave-1nl6"
server_key: "086f064ec1d4b39c7d60f3b44315d9b4"
server_key_iv: "259a9c38e028019e411488e8107ac4a4"
server_key_version: "encryptionkey@0"
I20260810 00:48:29.705716 17172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:48:29.720001 17172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:48:29.723820 17172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:48:29.724102 17172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:48:29.725003 17172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:48:29.726311 17172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:48:29.726415 17172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:29.726563 17172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:48:29.726636 17172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:29.761312 17172 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.2:46419
I20260810 00:48:29.761401 17367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.2:46419 every 8 connection(s)
I20260810 00:48:29.762619 17172 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
I20260810 00:48:29.762892 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 17172
I20260810 00:48:29.763144 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance
I20260810 00:48:29.763715 28696 external_mini_cluster.cc:1428] Setting key 22452c64ebfe99b6574ad99e693ff39e
Aug 10 00:48:29 dist-test-slave-1nl6 krb5kdc[16802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322909, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:48:29.787628 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:29.765598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:57291 (local address 127.28.6.62:36385)
0810 00:48:29.766217 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:29.766223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:29.767441 (+ 1218us) server_negotiation.cc:407] Connection header received
0810 00:48:29.770208 (+ 2767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:29.770212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:29.770277 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:29.770367 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:29.772536 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:29.773402 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:29.775871 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:29.776284 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:29.776387 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:29.780274 (+ 3887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:29.780282 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:29.780290 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:29.780335 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:29.784042 (+ 3707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:29.784820 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:29.784829 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:29.784836 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:29.785043 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:29.785525 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:29.785533 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:29.785538 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:29.785937 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:29.786092 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:29.786858 (+ 766us) server_negotiation.cc:299] Negotiation successful
0810 00:48:29.787135 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":370,"threads_started":1}
I20260810 00:48:29.788259 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:29.766364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.2:57291)
0810 00:48:29.767259 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:29.767315 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:29.769928 (+ 2613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:29.770639 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:29.770653 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:29.771473 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:48:29.772341 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:29.772369 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:29.773566 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:29.773573 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:29.775716 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:29.775747 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:29.776517 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:29.776527 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:29.776861 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:29.777568 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:48:29.777604 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:48:29.780017 (+ 2413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:48:29.784209 (+ 4192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:29.784215 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:29.784234 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:29.784666 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:29.785202 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:29.785205 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:29.785207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:29.785386 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:29.786104 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:48:29.786109 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:48:29.786479 (+ 370us) client_negotiation.cc:769] Sending connection context
0810 00:48:29.786704 (+ 225us) client_negotiation.cc:240] Negotiation successful
0810 00:48:29.787147 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":332,"threads_started":1}
I20260810 00:48:29.789758 17368 heartbeater.cc:347] Connected to a master server at 127.28.6.62:36385
I20260810 00:48:29.790078 17368 heartbeater.cc:464] Registering TS with master...
I20260810 00:48:29.790836 17368 heartbeater.cc:511] Master 127.28.6.62:36385 requested a full tablet report, sending...
I20260810 00:48:29.792460 16863 ts_manager.cc:194] Registered new tserver with Master: d3b6361ac1dd42839442096b8b99e033 (127.28.6.2:46419)
I20260810 00:48:29.793277 16863 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.2@KRBTEST.COM'} at 127.28.6.2:57291
WARNING: no policy specified for kudu/127.28.6.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab.
Entry for principal kudu/127.28.6.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab.
I20260810 00:48:29.827147 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.3:0
--local_ip_for_outbound_sockets=127.28.6.3
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:36385
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:44267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:48:30.168493 17376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:48:30.168753 17376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:48:30.168848 17376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:48:30.176236 17376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:48:30.176333 17376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:48:30.176438 17376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.3
I20260810 00:48:30.185803 17376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:44267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:36385
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.3
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:48:30.187126 17376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:48:30.188452 17376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:48:30.201347 17388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:30.201354 17389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:30.201646 17391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:48:30.202445 17376 server_base.cc:1034] running on GCE node
I20260810 00:48:30.203107 17376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:48:30.204118 17376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:48:30.205348 17376 hybrid_clock.cc:648] HybridClock initialized: now 1786322910205279 us; error 52 us; skew 500 ppm
I20260810 00:48:30.205642 17376 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:48:30 dist-test-slave-1nl6 krb5kdc[16802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322910, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:48:30.208649 17376 init.cc:377] Logged in from keytab as kudu/127.28.6.3@KRBTEST.COM (short username kudu)
I20260810 00:48:30.210248 17376 webserver.cc:460] Webserver started at http://127.28.6.3:46817/ using document root <none> and password file <none>
I20260810 00:48:30.210988 17376 fs_manager.cc:359] Metadata directory not provided
I20260810 00:48:30.211131 17376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:48:30.211432 17376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:48:30.214308 17376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/data/instance:
uuid: "e7878590bbf54ded8185416557742988"
format_stamp: "Formatted at 2026-08-10 00:48:30 on dist-test-slave-1nl6"
server_key: "56ea9e13eb293de681cbfcc4f50adb8a"
server_key_iv: "2ec817d455bd4df6613b559da0b18a6a"
server_key_version: "encryptionkey@0"
I20260810 00:48:30.215135 17376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance:
uuid: "e7878590bbf54ded8185416557742988"
format_stamp: "Formatted at 2026-08-10 00:48:30 on dist-test-slave-1nl6"
server_key: "56ea9e13eb293de681cbfcc4f50adb8a"
server_key_iv: "2ec817d455bd4df6613b559da0b18a6a"
server_key_version: "encryptionkey@0"
I20260810 00:48:30.220009 17376 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 00:48:30.223532 17402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:30.224759 17376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 00:48:30.224944 17376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/wal
uuid: "e7878590bbf54ded8185416557742988"
format_stamp: "Formatted at 2026-08-10 00:48:30 on dist-test-slave-1nl6"
server_key: "56ea9e13eb293de681cbfcc4f50adb8a"
server_key_iv: "2ec817d455bd4df6613b559da0b18a6a"
server_key_version: "encryptionkey@0"
I20260810 00:48:30.225188 17376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:48:30.247469 17376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:48:30.251434 17376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:48:30.251745 17376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:48:30.252743 17376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:48:30.254062 17376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:48:30.254140 17376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:30.254323 17376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:48:30.254376 17376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:30.262446 17166 heartbeater.cc:503] Master 127.28.6.62:36385 was elected leader, sending a full tablet report...
I20260810 00:48:30.289330 17376 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.3:39599
I20260810 00:48:30.289438 17589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.3:39599 every 8 connection(s)
I20260810 00:48:30.290624 17376 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
I20260810 00:48:30.291371 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 17376
I20260810 00:48:30.291601 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance
I20260810 00:48:30.292202 28696 external_mini_cluster.cc:1428] Setting key 7cc0b439c10317ccabe1d6eedf20f1a0
Aug 10 00:48:30 dist-test-slave-1nl6 krb5kdc[16802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322910, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:48:30.310962 17593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:30.293574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:52241 (local address 127.28.6.62:36385)
0810 00:48:30.294078 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:30.294087 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:30.295315 (+ 1228us) server_negotiation.cc:407] Connection header received
0810 00:48:30.297122 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:30.297126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:30.297189 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:30.297283 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:30.298635 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:30.299508 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:30.300954 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:30.301347 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:30.301438 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:30.304734 (+ 3296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:30.304739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:30.304742 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:30.304774 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:30.308137 (+ 3363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:30.308797 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:30.308801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:30.308803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:30.308918 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:30.309328 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:30.309331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:30.309333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:30.309670 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:30.309792 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:30.310425 (+ 633us) server_negotiation.cc:299] Negotiation successful
0810 00:48:30.310615 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":252,"threads_started":1}
I20260810 00:48:30.311883 17594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:30.294281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.3:52241)
0810 00:48:30.295157 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:30.295204 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:30.296889 (+ 1685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:30.297475 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:30.297482 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:30.297962 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:48:30.298495 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:30.298510 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:30.299642 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:30.299646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:30.300828 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:30.300841 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:30.301530 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:30.301535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:30.301734 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:30.302303 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:48:30.302328 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:48:30.304537 (+ 2209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:48:30.308255 (+ 3718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:30.308261 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:30.308272 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:30.308673 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:30.309042 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:30.309045 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:30.309048 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:30.309233 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:30.309781 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:48:30.309787 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:48:30.310070 (+ 283us) client_negotiation.cc:769] Sending connection context
0810 00:48:30.310297 (+ 227us) client_negotiation.cc:240] Negotiation successful
0810 00:48:30.310731 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":359,"threads_started":1}
I20260810 00:48:30.313282 17590 heartbeater.cc:347] Connected to a master server at 127.28.6.62:36385
I20260810 00:48:30.313548 17590 heartbeater.cc:464] Registering TS with master...
I20260810 00:48:30.314215 17590 heartbeater.cc:511] Master 127.28.6.62:36385 requested a full tablet report, sending...
I20260810 00:48:30.315621 16863 ts_manager.cc:194] Registered new tserver with Master: e7878590bbf54ded8185416557742988 (127.28.6.3:39599)
I20260810 00:48:30.316434 16863 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.3@KRBTEST.COM'} at 127.28.6.3:52241
I20260810 00:48:30.319983 28696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 00:48:30.336517 17593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:30.324555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54148 (local address 127.28.6.62:36385)
0810 00:48:30.324786 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:30.324793 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:30.325048 (+ 255us) server_negotiation.cc:407] Connection header received
0810 00:48:30.325167 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:30.325170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:30.325228 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:30.325306 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:30.326643 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:30.327398 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:30.328654 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:30.329070 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:30.329160 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:30.331645 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:30.331650 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:30.331653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:30.331700 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:30.334147 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:30.334761 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:30.334765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:30.334767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:30.334879 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:30.335256 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:30.335260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:30.335262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:30.335568 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:30.335753 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:30.336002 (+ 249us) server_negotiation.cc:299] Negotiation successful
0810 00:48:30.336192 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 00:48:30.340412 16863 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54148:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 00:48:30.347417 16863 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 00:48:30.389878 17615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:30.360648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44144 (local address 127.28.6.3:39599)
0810 00:48:30.380680 (+ 20032us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:30.380692 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:30.380948 (+ 256us) server_negotiation.cc:407] Connection header received
0810 00:48:30.381112 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:30.381123 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:30.381458 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:30.381665 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:48:30.382634 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:30.384172 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:30.387634 (+ 3462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:30.388651 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:30.388805 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:30.389079 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:30.389348 (+ 269us) server_negotiation.cc:299] Negotiation successful
0810 00:48:30.389515 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19811,"thread_start_us":20057,"threads_started":1}
I20260810 00:48:30.389945 17610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:30.359980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:39599 (local address 127.0.0.1:44144)
0810 00:48:30.380776 (+ 20796us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:30.380796 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:30.380934 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:30.381894 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:30.381901 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:30.381940 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:48:30.382353 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:30.382365 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:30.384299 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:30.384307 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:30.387109 (+ 2802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:30.387126 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:30.388851 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:30.388858 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:30.389075 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:30.389141 (+ 66us) client_negotiation.cc:769] Sending connection context
0810 00:48:30.389301 (+ 160us) client_negotiation.cc:240] Negotiation successful
0810 00:48:30.389505 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20645,"mutex_wait_us":57,"thread_start_us":19159,"threads_started":1}
I20260810 00:48:30.390494 17611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:30.360333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53294 (local address 127.28.6.2:46419)
0810 00:48:30.381877 (+ 21544us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:30.381884 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:30.381909 (+ 25us) server_negotiation.cc:407] Connection header received
0810 00:48:30.382011 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:30.382019 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:30.382221 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:30.382422 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:48:30.383218 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:30.385075 (+ 1857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:30.387400 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:30.388804 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:30.388940 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:30.389130 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:30.389977 (+ 847us) server_negotiation.cc:299] Negotiation successful
0810 00:48:30.390135 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21397,"thread_start_us":21264,"threads_started":1}
I20260810 00:48:30.390805 17609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:30.359733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:46419 (local address 127.0.0.1:53294)
0810 00:48:30.379790 (+ 20057us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:30.379814 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:30.379968 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:30.382465 (+ 2497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:30.382472 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:30.382505 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:48:30.383021 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:30.383038 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:30.385206 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:30.385212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:30.387184 (+ 1972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:30.387204 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:30.389358 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:30.389366 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:30.389593 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:30.389746 (+ 153us) client_negotiation.cc:769] Sending connection context
0810 00:48:30.389837 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 00:48:30.390395 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19883,"mutex_wait_us":544,"thread_start_us":19726,"threads_started":1}
I20260810 00:48:30.391559 17612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:30.360540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46318 (local address 127.28.6.1:39615)
0810 00:48:30.381581 (+ 21041us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:30.381591 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:30.381621 (+ 30us) server_negotiation.cc:407] Connection header received
0810 00:48:30.381718 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:30.381724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:30.381957 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:30.382149 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:48:30.382855 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:30.384512 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:30.387304 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:30.388797 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:30.388940 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:30.389166 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:30.391090 (+ 1924us) server_negotiation.cc:299] Negotiation successful
0810 00:48:30.391263 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20852,"thread_start_us":20666,"threads_started":1}
I20260810 00:48:30.394027 17265 tablet_service.cc:1467] Processing CreateTablet for tablet 42e37ba6bf01415dbef284bbd487bb8a (DEFAULT_TABLE table=test-table [id=f71c9f82f47c4d579f96566c9ce18787]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:48:30.394029 17486 tablet_service.cc:1467] Processing CreateTablet for tablet 42e37ba6bf01415dbef284bbd487bb8a (DEFAULT_TABLE table=test-table [id=f71c9f82f47c4d579f96566c9ce18787]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:48:30.395270 17608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:30.359423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:39615 (local address 127.0.0.1:46318)
0810 00:48:30.376108 (+ 16685us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:30.376154 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:30.376332 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:30.382158 (+ 5826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:30.382168 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:30.382192 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:48:30.382649 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:30.382660 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:30.384676 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:30.384681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:30.387147 (+ 2466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:30.387160 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:30.389374 (+ 2214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:30.389379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:30.389594 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:30.389747 (+ 153us) client_negotiation.cc:769] Sending connection context
0810 00:48:30.389845 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 00:48:30.389995 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16453,"mutex_wait_us":264,"thread_start_us":16286,"threads_started":1}
I20260810 00:48:30.395357 17265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42e37ba6bf01415dbef284bbd487bb8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:48:30.395364 17486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42e37ba6bf01415dbef284bbd487bb8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:48:30.399437 17064 tablet_service.cc:1467] Processing CreateTablet for tablet 42e37ba6bf01415dbef284bbd487bb8a (DEFAULT_TABLE table=test-table [id=f71c9f82f47c4d579f96566c9ce18787]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:48:30.400859 17064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42e37ba6bf01415dbef284bbd487bb8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:48:30.407028 17617 tablet_bootstrap.cc:492] T 42e37ba6bf01415dbef284bbd487bb8a P d3b6361ac1dd42839442096b8b99e033: Bootstrap starting.
I20260810 00:48:30.410334 17617 tablet_bootstrap.cc:654] T 42e37ba6bf01415dbef284bbd487bb8a P d3b6361ac1dd42839442096b8b99e033: Neither blocks nor log segments found. Creating new log.
I20260810 00:48:30.411561 17617 log.cc:824] T 42e37ba6bf01415dbef284bbd487bb8a P d3b6361ac1dd42839442096b8b99e033: Log is configured to *not* fsync() on all Append() calls
I20260810 00:48:30.411669 17618 tablet_bootstrap.cc:492] T 42e37ba6bf01415dbef284bbd487bb8a P e7878590bbf54ded8185416557742988: Bootstrap starting.
I20260810 00:48:30.415251 17624 tablet_bootstrap.cc:492] T 42e37ba6bf01415dbef284bbd487bb8a P 5baea45d390a4f0ab315bcf66cf1687a: Bootstrap starting.
I20260810 00:48:30.416152 17618 tablet_bootstrap.cc:654] T 42e37ba6bf01415dbef284bbd487bb8a P e7878590bbf54ded8185416557742988: Neither blocks nor log segments found. Creating new log.
I20260810 00:48:30.417337 17618 log.cc:824] T 42e37ba6bf01415dbef284bbd487bb8a P e7878590bbf54ded8185416557742988: Log is configured to *not* fsync() on all Append() calls
I20260810 00:48:30.418509 17624 tablet_bootstrap.cc:654] T 42e37ba6bf01415dbef284bbd487bb8a P 5baea45d390a4f0ab315bcf66cf1687a: Neither blocks nor log segments found. Creating new log.
I20260810 00:48:30.418567 17617 tablet_bootstrap.cc:492] T 42e37ba6bf01415dbef284bbd487bb8a P d3b6361ac1dd42839442096b8b99e033: No bootstrap required, opened a new log
I20260810 00:48:30.418947 17617 ts_tablet_manager.cc:1397] T 42e37ba6bf01415dbef284bbd487bb8a P d3b6361ac1dd42839442096b8b99e033: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260810 00:48:30.419605 17624 log.cc:824] T 42e37ba6bf01415dbef284bbd487bb8a P 5baea45d390a4f0ab315bcf66cf1687a: Log is configured to *not* fsync() on all Append() calls
I20260810 00:48:30.420527 17618 tablet_bootstrap.cc:492] T 42e37ba6bf01415dbef284bbd487bb8a P e7878590bbf54ded8185416557742988: No bootstrap required, opened a new log
I20260810 00:48:30.420801 17618 ts_tablet_manager.cc:1397] T 42e37ba6bf01415dbef284bbd487bb8a P e7878590bbf54ded8185416557742988: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260810 00:48:30.422257 17624 tablet_bootstrap.cc:492] T 42e37ba6bf01415dbef284bbd487bb8a P 5baea45d390a4f0ab315bcf66cf1687a: No bootstrap required, opened a new log
I20260810 00:48:30.422519 17624 ts_tablet_manager.cc:1397] T 42e37ba6bf01415dbef284bbd487bb8a P 5baea45d390a4f0ab315bcf66cf1687a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 00:48:30.429499 17617 raft_consensus.cc:348] T 42e37ba6bf01415dbef284bbd487bb8a P d3b6361ac1dd42839442096b8b99e033 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5baea45d390a4f0ab315bcf66cf1687a" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 39615 } } peers { permanent_uuid: "d3b6361ac1dd42839442096b8b99e033" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46419 } } peers { permanent_uuid: "e7878590bbf54ded8185416557742988" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39599 } }
I20260810 00:48:30.429948 17617 raft_consensus.cc:374] T 42e37ba6bf01415dbef284bbd487bb8a P d3b6361ac1dd42839442096b8b99e033 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:48:30.430054 17617 raft_consensus.cc:729] T 42e37ba6bf01415dbef284bbd487bb8a P d3b6361ac1dd42839442096b8b99e033 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3b6361ac1dd42839442096b8b99e033, State: Initialized, Role: FOLLOWER
I20260810 00:48:30.430418 17624 raft_consensus.cc:348] T 42e37ba6bf01415dbef284bbd487bb8a P 5baea45d390a4f0ab315bcf66cf1687a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5baea45d390a4f0ab315bcf66cf1687a" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 39615 } } peers { permanent_uuid: "d3b6361ac1dd42839442096b8b99e033" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46419 } } peers { permanent_uuid: "e7878590bbf54ded8185416557742988" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39599 } }
I20260810 00:48:30.430781 17624 raft_consensus.cc:374] T 42e37ba6bf01415dbef284bbd487bb8a P 5baea45d390a4f0ab315bcf66cf1687a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:48:30.430676 17617 consensus_queue.cc:260] T 42e37ba6bf01415dbef284bbd487bb8a P d3b6361ac1dd42839442096b8b99e033 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5baea45d390a4f0ab315bcf66cf1687a" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 39615 } } peers { permanent_uuid: "d3b6361ac1dd42839442096b8b99e033" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46419 } } peers { permanent_uuid: "e7878590bbf54ded8185416557742988" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39599 } }
I20260810 00:48:30.430891 17624 raft_consensus.cc:729] T 42e37ba6bf01415dbef284bbd487bb8a P 5baea45d390a4f0ab315bcf66cf1687a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5baea45d390a4f0ab315bcf66cf1687a, State: Initialized, Role: FOLLOWER
I20260810 00:48:30.431454 17624 consensus_queue.cc:260] T 42e37ba6bf01415dbef284bbd487bb8a P 5baea45d390a4f0ab315bcf66cf1687a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5baea45d390a4f0ab315bcf66cf1687a" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 39615 } } peers { permanent_uuid: "d3b6361ac1dd42839442096b8b99e033" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46419 } } peers { permanent_uuid: "e7878590bbf54ded8185416557742988" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39599 } }
I20260810 00:48:30.431247 17618 raft_consensus.cc:348] T 42e37ba6bf01415dbef284bbd487bb8a P e7878590bbf54ded8185416557742988 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5baea45d390a4f0ab315bcf66cf1687a" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 39615 } } peers { permanent_uuid: "d3b6361ac1dd42839442096b8b99e033" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46419 } } peers { permanent_uuid: "e7878590bbf54ded8185416557742988" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39599 } }
I20260810 00:48:30.431728 17618 raft_consensus.cc:374] T 42e37ba6bf01415dbef284bbd487bb8a P e7878590bbf54ded8185416557742988 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:48:30.431851 17618 raft_consensus.cc:729] T 42e37ba6bf01415dbef284bbd487bb8a P e7878590bbf54ded8185416557742988 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7878590bbf54ded8185416557742988, State: Initialized, Role: FOLLOWER
I20260810 00:48:30.432571 17618 consensus_queue.cc:260] T 42e37ba6bf01415dbef284bbd487bb8a P e7878590bbf54ded8185416557742988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5baea45d390a4f0ab315bcf66cf1687a" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 39615 } } peers { permanent_uuid: "d3b6361ac1dd42839442096b8b99e033" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46419 } } peers { permanent_uuid: "e7878590bbf54ded8185416557742988" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39599 } }
I20260810 00:48:30.433964 17590 heartbeater.cc:503] Master 127.28.6.62:36385 was elected leader, sending a full tablet report...
I20260810 00:48:30.434744 17618 ts_tablet_manager.cc:1428] T 42e37ba6bf01415dbef284bbd487bb8a P e7878590bbf54ded8185416557742988: Time spent starting tablet: real 0.014s user 0.008s sys 0.005s
I20260810 00:48:30.437849 17368 heartbeater.cc:503] Master 127.28.6.62:36385 was elected leader, sending a full tablet report...
I20260810 00:48:30.438700 17617 ts_tablet_manager.cc:1428] T 42e37ba6bf01415dbef284bbd487bb8a P d3b6361ac1dd42839442096b8b99e033: Time spent starting tablet: real 0.020s user 0.014s sys 0.004s
I20260810 00:48:30.441819 17624 ts_tablet_manager.cc:1428] T 42e37ba6bf01415dbef284bbd487bb8a P 5baea45d390a4f0ab315bcf66cf1687a: Time spent starting tablet: real 0.019s user 0.014s sys 0.003s
W20260810 00:48:30.484365 17167 tablet.cc:2369] T 42e37ba6bf01415dbef284bbd487bb8a P 5baea45d390a4f0ab315bcf66cf1687a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 00:48:30.515075 17369 tablet.cc:2369] T 42e37ba6bf01415dbef284bbd487bb8a P d3b6361ac1dd42839442096b8b99e033: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 00:48:30.542503 17591 tablet.cc:2369] T 42e37ba6bf01415dbef284bbd487bb8a P e7878590bbf54ded8185416557742988: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:48:30.588157 17631 raft_consensus.cc:482] T 42e37ba6bf01415dbef284bbd487bb8a P e7878590bbf54ded8185416557742988 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:48:30.588501 17631 raft_consensus.cc:504] T 42e37ba6bf01415dbef284bbd487bb8a P e7878590bbf54ded8185416557742988 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5baea45d390a4f0ab315bcf66cf1687a" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 39615 } } peers { permanent_uuid: "d3b6361ac1dd42839442096b8b99e033" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46419 } } peers { permanent_uuid: "e7878590bbf54ded8185416557742988" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39599 } }
I20260810 00:48:30.590319 17631 leader_election.cc:288] T 42e37ba6bf01415dbef284bbd487bb8a P e7878590bbf54ded8185416557742988 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5baea45d390a4f0ab315bcf66cf1687a (127.28.6.1:39615), d3b6361ac1dd42839442096b8b99e033 (127.28.6.2:46419)
I20260810 00:48:30.596856 17594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:30.590603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:46419 (local address 127.28.6.3:47155)
0810 00:48:30.590842 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:30.590866 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:30.591063 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:30.591583 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:30.591588 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:30.591623 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:48:30.591874 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:30.591886 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:30.593225 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:30.593231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:30.594680 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:30.594694 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:30.596133 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:30.596139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:30.596323 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:30.596346 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 00:48:30.596456 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 00:48:30.596583 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260810 00:48:30.596858 17611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:30.591099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:47155 (local address 127.28.6.2:46419)
0810 00:48:30.591306 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:30.591310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:30.591325 (+ 15us) server_negotiation.cc:407] Connection header received
0810 00:48:30.591386 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:30.591389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:30.591445 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:30.591555 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:48:30.592035 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:30.593042 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:30.594848 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:30.595963 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:30.596110 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:30.596202 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:30.596485 (+ 283us) server_negotiation.cc:299] Negotiation successful
0810 00:48:30.596604 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 00:48:30.597769 17633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:30.590717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:39615 (local address 127.28.6.3:45145)
0810 00:48:30.591649 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:30.591663 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:30.591778 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:30.592132 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:30.592136 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:30.592176 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:48:30.592556 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:30.592569 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:30.593847 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:30.593851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:30.595469 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:30.595486 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:30.597039 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:30.597045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:30.597209 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:30.597265 (+ 56us) client_negotiation.cc:769] Sending connection context
0810 00:48:30.597369 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 00:48:30.597491 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"thread_start_us":579,"threads_started":1}
I20260810 00:48:30.597769 17612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:30.590794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:45145 (local address 127.28.6.1:39615)
0810 00:48:30.591092 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:30.591101 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:30.591758 (+ 657us) server_negotiation.cc:407] Connection header received
0810 00:48:30.591907 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:30.591913 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:30.592004 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:30.592083 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:48:30.592727 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:30.593655 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:30.595622 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:30.596672 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:30.596816 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:30.596914 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:30.597413 (+ 499us) server_negotiation.cc:299] Negotiation successful
0810 00:48:30.597507 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260810 00:48:30.598169 17297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42e37ba6bf01415dbef284bbd487bb8a" candidate_uuid: "e7878590bbf54ded8185416557742988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3b6361ac1dd42839442096b8b99e033" is_pre_election: true
I20260810 00:48:30.598783 17297 raft_consensus.cc:2454] T 42e37ba6bf01415dbef284bbd487bb8a P d3b6361ac1dd42839442096b8b99e033 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7878590bbf54ded8185416557742988 in term 0.
I20260810 00:48:30.598850 17095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42e37ba6bf01415dbef284bbd487bb8a" candidate_uuid: "e7878590bbf54ded8185416557742988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5baea45d390a4f0ab315bcf66cf1687a" is_pre_election: true
I20260810 00:48:30.599390 17095 raft_consensus.cc:2454] T 42e37ba6bf01415dbef284bbd487bb8a P 5baea45d390a4f0ab315bcf66cf1687a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7878590bbf54ded8185416557742988 in term 0.
I20260810 00:48:30.599483 17404 leader_election.cc:302] T 42e37ba6bf01415dbef284bbd487bb8a P e7878590bbf54ded8185416557742988 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3b6361ac1dd42839442096b8b99e033, e7878590bbf54ded8185416557742988; no voters:
I20260810 00:48:30.599931 17631 raft_consensus.cc:2781] T 42e37ba6bf01415dbef284bbd487bb8a P e7878590bbf54ded8185416557742988 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 00:48:30.600097 17631 raft_consensus.cc:482] T 42e37ba6bf01415dbef284bbd487bb8a P e7878590bbf54ded8185416557742988 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 00:48:30.600219 17631 raft_consensus.cc:3037] T 42e37ba6bf01415dbef284bbd487bb8a P e7878590bbf54ded8185416557742988 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:48:30.602385 17631 raft_consensus.cc:504] T 42e37ba6bf01415dbef284bbd487bb8a P e7878590bbf54ded8185416557742988 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5baea45d390a4f0ab315bcf66cf1687a" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 39615 } } peers { permanent_uuid: "d3b6361ac1dd42839442096b8b99e033" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46419 } } peers { permanent_uuid: "e7878590bbf54ded8185416557742988" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39599 } }
I20260810 00:48:30.603006 17631 leader_election.cc:288] T 42e37ba6bf01415dbef284bbd487bb8a P e7878590bbf54ded8185416557742988 [CANDIDATE]: Term 1 election: Requested vote from peers 5baea45d390a4f0ab315bcf66cf1687a (127.28.6.1:39615), d3b6361ac1dd42839442096b8b99e033 (127.28.6.2:46419)
I20260810 00:48:30.603374 17095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42e37ba6bf01415dbef284bbd487bb8a" candidate_uuid: "e7878590bbf54ded8185416557742988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5baea45d390a4f0ab315bcf66cf1687a"
I20260810 00:48:30.603564 17095 raft_consensus.cc:3037] T 42e37ba6bf01415dbef284bbd487bb8a P 5baea45d390a4f0ab315bcf66cf1687a [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:48:30.603535 17297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42e37ba6bf01415dbef284bbd487bb8a" candidate_uuid: "e7878590bbf54ded8185416557742988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3b6361ac1dd42839442096b8b99e033"
I20260810 00:48:30.603750 17297 raft_consensus.cc:3037] T 42e37ba6bf01415dbef284bbd487bb8a P d3b6361ac1dd42839442096b8b99e033 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:48:30.605746 17095 raft_consensus.cc:2454] T 42e37ba6bf01415dbef284bbd487bb8a P 5baea45d390a4f0ab315bcf66cf1687a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7878590bbf54ded8185416557742988 in term 1.
I20260810 00:48:30.605979 17297 raft_consensus.cc:2454] T 42e37ba6bf01415dbef284bbd487bb8a P d3b6361ac1dd42839442096b8b99e033 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7878590bbf54ded8185416557742988 in term 1.
I20260810 00:48:30.606197 17408 leader_election.cc:302] T 42e37ba6bf01415dbef284bbd487bb8a P e7878590bbf54ded8185416557742988 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5baea45d390a4f0ab315bcf66cf1687a, e7878590bbf54ded8185416557742988; no voters:
I20260810 00:48:30.606526 17631 raft_consensus.cc:2781] T 42e37ba6bf01415dbef284bbd487bb8a P e7878590bbf54ded8185416557742988 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:48:30.607046 17631 raft_consensus.cc:686] T 42e37ba6bf01415dbef284bbd487bb8a P e7878590bbf54ded8185416557742988 [term 1 LEADER]: Becoming Leader. State: Replica: e7878590bbf54ded8185416557742988, State: Running, Role: LEADER
I20260810 00:48:30.607585 17631 consensus_queue.cc:237] T 42e37ba6bf01415dbef284bbd487bb8a P e7878590bbf54ded8185416557742988 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5baea45d390a4f0ab315bcf66cf1687a" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 39615 } } peers { permanent_uuid: "d3b6361ac1dd42839442096b8b99e033" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46419 } } peers { permanent_uuid: "e7878590bbf54ded8185416557742988" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39599 } }
I20260810 00:48:30.612432 16863 catalog_manager.cc:5495] T 42e37ba6bf01415dbef284bbd487bb8a P e7878590bbf54ded8185416557742988 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7878590bbf54ded8185416557742988 (127.28.6.3). New cstate: current_term: 1 leader_uuid: "e7878590bbf54ded8185416557742988" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5baea45d390a4f0ab315bcf66cf1687a" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 39615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3b6361ac1dd42839442096b8b99e033" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7878590bbf54ded8185416557742988" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39599 } health_report { overall_health: HEALTHY } } }
I20260810 00:48:30.680497 17615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:30.675336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44154 (local address 127.28.6.3:39599)
0810 00:48:30.675540 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:30.675545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:30.675562 (+ 17us) server_negotiation.cc:407] Connection header received
0810 00:48:30.675656 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:30.675660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:30.675766 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:30.675895 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:48:30.676307 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:30.677048 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:30.678409 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:30.678793 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:30.678885 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:30.679410 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:48:30.679933 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:48:30.680042 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:30.680141 (+ 99us) server_negotiation.cc:299] Negotiation successful
0810 00:48:30.680205 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 00:48:30.688745 17095 raft_consensus.cc:1261] T 42e37ba6bf01415dbef284bbd487bb8a P 5baea45d390a4f0ab315bcf66cf1687a [term 1 FOLLOWER]: Refusing update from remote peer e7878590bbf54ded8185416557742988: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:48:30.688735 17297 raft_consensus.cc:1261] T 42e37ba6bf01415dbef284bbd487bb8a P d3b6361ac1dd42839442096b8b99e033 [term 1 FOLLOWER]: Refusing update from remote peer e7878590bbf54ded8185416557742988: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:48:30.689867 17634 consensus_queue.cc:1035] T 42e37ba6bf01415dbef284bbd487bb8a P e7878590bbf54ded8185416557742988 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3b6361ac1dd42839442096b8b99e033" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:48:30.690237 17631 consensus_queue.cc:1035] T 42e37ba6bf01415dbef284bbd487bb8a P e7878590bbf54ded8185416557742988 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5baea45d390a4f0ab315bcf66cf1687a" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 39615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:48:30.703783 17638 mvcc.cc:205] Tried to move back new op lower bound from 7316778642170167296 to 7316778641852801024. Current Snapshot: MvccSnapshot[applied={T|T < 7316778642170167296}]
I20260810 00:48:30.707614 17639 mvcc.cc:205] Tried to move back new op lower bound from 7316778642170167296 to 7316778641852801024. Current Snapshot: MvccSnapshot[applied={T|T < 7316778642170167296}]
I20260810 00:48:30.717072 16863 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:48:30.717331 16863 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:48:30.720743 16863 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d: Sending DeleteTablet for 3 replicas of tablet 42e37ba6bf01415dbef284bbd487bb8a
I20260810 00:48:30.721971 17265 tablet_service.cc:1514] Processing DeleteTablet for tablet 42e37ba6bf01415dbef284bbd487bb8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:48:30 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:53294
I20260810 00:48:30.721968 17064 tablet_service.cc:1514] Processing DeleteTablet for tablet 42e37ba6bf01415dbef284bbd487bb8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:48:30 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:46318
I20260810 00:48:30.722191 17486 tablet_service.cc:1514] Processing DeleteTablet for tablet 42e37ba6bf01415dbef284bbd487bb8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:48:30 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:44144
I20260810 00:48:30.723282 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 16969
I20260810 00:48:30.731386 17655 tablet_replica.cc:330] T 42e37ba6bf01415dbef284bbd487bb8a P 5baea45d390a4f0ab315bcf66cf1687a: stopping tablet replica
I20260810 00:48:30.731900 17655 raft_consensus.cc:2229] T 42e37ba6bf01415dbef284bbd487bb8a P 5baea45d390a4f0ab315bcf66cf1687a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:48:30.732502 17655 raft_consensus.cc:2258] T 42e37ba6bf01415dbef284bbd487bb8a P 5baea45d390a4f0ab315bcf66cf1687a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:48:30.735288 17655 ts_tablet_manager.cc:1905] T 42e37ba6bf01415dbef284bbd487bb8a P 5baea45d390a4f0ab315bcf66cf1687a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:48:30.743551 17656 tablet_replica.cc:330] T 42e37ba6bf01415dbef284bbd487bb8a P e7878590bbf54ded8185416557742988: stopping tablet replica
I20260810 00:48:30.744107 17656 raft_consensus.cc:2229] T 42e37ba6bf01415dbef284bbd487bb8a P e7878590bbf54ded8185416557742988 [term 1 LEADER]: Raft consensus shutting down.
I20260810 00:48:30.744930 17656 raft_consensus.cc:2258] T 42e37ba6bf01415dbef284bbd487bb8a P e7878590bbf54ded8185416557742988 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:48:30.745823 17651 tablet_replica.cc:330] T 42e37ba6bf01415dbef284bbd487bb8a P d3b6361ac1dd42839442096b8b99e033: stopping tablet replica
I20260810 00:48:30.746230 17651 raft_consensus.cc:2229] T 42e37ba6bf01415dbef284bbd487bb8a P d3b6361ac1dd42839442096b8b99e033 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:48:30.746716 17651 raft_consensus.cc:2258] T 42e37ba6bf01415dbef284bbd487bb8a P d3b6361ac1dd42839442096b8b99e033 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:48:30.747305 17655 ts_tablet_manager.cc:1918] T 42e37ba6bf01415dbef284bbd487bb8a P 5baea45d390a4f0ab315bcf66cf1687a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:48:30.747539 17655 log.cc:1196] T 42e37ba6bf01415dbef284bbd487bb8a P 5baea45d390a4f0ab315bcf66cf1687a: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/wal/wals/42e37ba6bf01415dbef284bbd487bb8a
I20260810 00:48:30.748260 17655 ts_tablet_manager.cc:1939] T 42e37ba6bf01415dbef284bbd487bb8a P 5baea45d390a4f0ab315bcf66cf1687a: Deleting consensus metadata
I20260810 00:48:30.749094 17651 ts_tablet_manager.cc:1905] T 42e37ba6bf01415dbef284bbd487bb8a P d3b6361ac1dd42839442096b8b99e033: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:48:30.750279 16848 catalog_manager.cc:4841] TS 5baea45d390a4f0ab315bcf66cf1687a (127.28.6.1:39615): tablet 42e37ba6bf01415dbef284bbd487bb8a (table test-table [id=f71c9f82f47c4d579f96566c9ce18787]) successfully deleted
I20260810 00:48:30.752034 17656 ts_tablet_manager.cc:1905] T 42e37ba6bf01415dbef284bbd487bb8a P e7878590bbf54ded8185416557742988: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:48:30.756362 17651 ts_tablet_manager.cc:1918] T 42e37ba6bf01415dbef284bbd487bb8a P d3b6361ac1dd42839442096b8b99e033: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:48:30.756549 17651 log.cc:1196] T 42e37ba6bf01415dbef284bbd487bb8a P d3b6361ac1dd42839442096b8b99e033: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/wal/wals/42e37ba6bf01415dbef284bbd487bb8a
Aug 10 00:48:30 dist-test-slave-1nl6 krb5kdc[16802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.1@KRBTEST.COM
I20260810 00:48:30.757092 17651 ts_tablet_manager.cc:1939] T 42e37ba6bf01415dbef284bbd487bb8a P d3b6361ac1dd42839442096b8b99e033: Deleting consensus metadata
I20260810 00:48:30.758924 16842 catalog_manager.cc:4841] TS d3b6361ac1dd42839442096b8b99e033 (127.28.6.2:46419): tablet 42e37ba6bf01415dbef284bbd487bb8a (table test-table [id=f71c9f82f47c4d579f96566c9ce18787]) successfully deleted
I20260810 00:48:30.759776 17656 ts_tablet_manager.cc:1918] T 42e37ba6bf01415dbef284bbd487bb8a P e7878590bbf54ded8185416557742988: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:48:30.759958 17656 log.cc:1196] T 42e37ba6bf01415dbef284bbd487bb8a P e7878590bbf54ded8185416557742988: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/wal/wals/42e37ba6bf01415dbef284bbd487bb8a
I20260810 00:48:30.760538 17656 ts_tablet_manager.cc:1939] T 42e37ba6bf01415dbef284bbd487bb8a P e7878590bbf54ded8185416557742988: Deleting consensus metadata
I20260810 00:48:30.762267 16845 catalog_manager.cc:4841] TS e7878590bbf54ded8185416557742988 (127.28.6.3:39599): tablet 42e37ba6bf01415dbef284bbd487bb8a (table test-table [id=f71c9f82f47c4d579f96566c9ce18787]) successfully deleted
I20260810 00:48:30.767354 17612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:30.724528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46326 (local address 127.28.6.1:39615)
0810 00:48:30.724708 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:30.724715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:30.744750 (+ 20035us) server_negotiation.cc:407] Connection header received
0810 00:48:30.745251 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:30.745257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:30.745354 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:30.745431 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:30.747539 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:30.748726 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:30.751480 (+ 2754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:30.752136 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:30.752278 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:30.758229 (+ 5951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:30.758245 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:30.758263 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:30.758305 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:30.763858 (+ 5553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:30.764812 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:30.764822 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:30.764826 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:30.764980 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:30.765444 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:30.765449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:30.765453 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:30.765981 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:30.766262 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:30.766655 (+ 393us) server_negotiation.cc:299] Negotiation successful
0810 00:48:30.766978 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 00:48:30.768682 17161 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:48:30.863487 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 16969
I20260810 00:48:30.893728 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 17172
Aug 10 00:48:30 dist-test-slave-1nl6 krb5kdc[16802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.2@KRBTEST.COM
I20260810 00:48:30.915726 17611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:30.894607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53304 (local address 127.28.6.2:46419)
0810 00:48:30.894794 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:30.894799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:30.894821 (+ 22us) server_negotiation.cc:407] Connection header received
0810 00:48:30.895071 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:30.895076 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:30.895142 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:30.895239 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:30.896940 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:30.897682 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:30.899584 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:30.899970 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:30.900067 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:30.909783 (+ 9716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:30.909798 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:30.909814 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:30.909848 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:30.912930 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:30.913619 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:30.913627 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:30.913629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:30.913754 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:30.914145 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:30.914150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:30.914152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:30.914524 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:30.914825 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:30.915158 (+ 333us) server_negotiation.cc:299] Negotiation successful
0810 00:48:30.915395 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 00:48:30.916427 17359 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:48:31.007558 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 17172
I20260810 00:48:31.037915 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 17376
Aug 10 00:48:31 dist-test-slave-1nl6 krb5kdc[16802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.3@KRBTEST.COM
I20260810 00:48:31.054275 17615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:31.038739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44168 (local address 127.28.6.3:39599)
0810 00:48:31.038969 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:31.038976 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:31.039083 (+ 107us) server_negotiation.cc:407] Connection header received
0810 00:48:31.039317 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:31.039322 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:31.039393 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:31.039513 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:31.040779 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:31.041691 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:31.042996 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:31.043432 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:31.043593 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:31.048006 (+ 4413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:31.048020 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:31.048038 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:31.048084 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:31.051309 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:31.051987 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:31.052000 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:31.052002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:31.052187 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:31.052598 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:31.052603 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:31.052614 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:31.052963 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:31.053326 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:31.053609 (+ 283us) server_negotiation.cc:299] Negotiation successful
0810 00:48:31.053851 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 00:48:31.055070 17583 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:48:31.179096 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 17376
I20260810 00:48:31.206698 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 16821
I20260810 00:48:31.207907 16913 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:48:31.315500 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 16821
I20260810 00:48:31.346280 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:36385
--webserver_interface=127.28.6.62
--webserver_port=43747
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:44267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:36385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:48:31.681622 17662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:48:31.681937 17662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:48:31.682126 17662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:48:31.689011 17662 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:48:31.689164 17662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:48:31.689280 17662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:48:31.689379 17662 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:48:31.689468 17662 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:48:31.699220 17662 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:44267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:36385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:36385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=43747
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:48:31.700579 17662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:48:31.701930 17662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:48:31.727392 17679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:31.727367 17677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:31.727361 17675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:48:31.728341 17662 server_base.cc:1034] running on GCE node
I20260810 00:48:31.729115 17662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:48:31.730626 17662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:48:31.731838 17662 hybrid_clock.cc:648] HybridClock initialized: now 1786322911731754 us; error 71 us; skew 500 ppm
I20260810 00:48:31.732270 17662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:48:31 dist-test-slave-1nl6 krb5kdc[16802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322911, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:48:31.736059 17662 init.cc:377] Logged in from keytab as kudu/127.28.6.62@KRBTEST.COM (short username kudu)
I20260810 00:48:31.737632 17662 webserver.cc:460] Webserver started at http://127.28.6.62:43747/ using document root <none> and password file <none>
I20260810 00:48:31.738453 17662 fs_manager.cc:359] Metadata directory not provided
I20260810 00:48:31.738621 17662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:48:31.744525 17662 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260810 00:48:31.747452 17688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:31.748755 17662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 00:48:31.748950 17662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "4fd535fd266f44949f3ea71c1aefee5d"
format_stamp: "Formatted at 2026-08-10 00:48:28 on dist-test-slave-1nl6"
server_key: "2171d19594674b98514a47a2cade09e3"
server_key_iv: "50b9e3e15cbf1da654407d5cac7b55d8"
server_key_version: "encryptionkey@0"
I20260810 00:48:31.750028 17662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:48:31.773998 17662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:48:31.778506 17662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:48:31.778801 17662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:48:31.797821 17662 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:36385
I20260810 00:48:31.797840 17771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:36385 every 8 connection(s)
I20260810 00:48:31.799335 17662 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:48:31.801383 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 17662
I20260810 00:48:31.802150 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.1:39615
--local_ip_for_outbound_sockets=127.28.6.1
--tserver_master_addrs=127.28.6.62:36385
--webserver_port=42113
--webserver_interface=127.28.6.1
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:44267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 00:48:31.806350 17773 sys_catalog.cc:263] Verifying existing consensus state
I20260810 00:48:31.809628 17773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d: Bootstrap starting.
I20260810 00:48:31.829378 17773 log.cc:824] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d: Log is configured to *not* fsync() on all Append() calls
I20260810 00:48:31.841447 17773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 00:48:31.842013 17773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d: Bootstrap complete.
I20260810 00:48:31.851126 17773 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd535fd266f44949f3ea71c1aefee5d" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36385 } }
I20260810 00:48:31.851941 17773 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd535fd266f44949f3ea71c1aefee5d, State: Initialized, Role: FOLLOWER
I20260810 00:48:31.852846 17773 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd535fd266f44949f3ea71c1aefee5d" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36385 } }
I20260810 00:48:31.853121 17773 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:48:31.853302 17773 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:48:31.853510 17773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [term 1 FOLLOWER]: Advancing to term 2
I20260810 00:48:31.857535 17773 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd535fd266f44949f3ea71c1aefee5d" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36385 } }
I20260810 00:48:31.858125 17773 leader_election.cc:302] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fd535fd266f44949f3ea71c1aefee5d; no voters:
I20260810 00:48:31.858793 17773 leader_election.cc:288] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 00:48:31.858950 17784 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [term 2 FOLLOWER]: Leader election won for term 2
I20260810 00:48:31.861328 17773 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:48:31.863085 17784 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [term 2 LEADER]: Becoming Leader. State: Replica: 4fd535fd266f44949f3ea71c1aefee5d, State: Running, Role: LEADER
I20260810 00:48:31.863759 17784 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd535fd266f44949f3ea71c1aefee5d" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36385 } }
I20260810 00:48:31.866580 17788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fd535fd266f44949f3ea71c1aefee5d. Latest consensus state: current_term: 2 leader_uuid: "4fd535fd266f44949f3ea71c1aefee5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd535fd266f44949f3ea71c1aefee5d" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36385 } } }
I20260810 00:48:31.867079 17788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [sys.catalog]: This master's current role is: LEADER
I20260810 00:48:31.867704 17785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4fd535fd266f44949f3ea71c1aefee5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd535fd266f44949f3ea71c1aefee5d" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36385 } } }
I20260810 00:48:31.868036 17785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [sys.catalog]: This master's current role is: LEADER
I20260810 00:48:31.871676 17800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:48:31.887640 17800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:48:31.889528 17800 catalog_manager.cc:1259] Loaded cluster ID: 5bee55c23a1e4e328058c3e0186cb203
I20260810 00:48:31.889650 17800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:48:31.893838 17800 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260810 00:48:31.919450 17800 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260810 00:48:31.929993 17800 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:48:31.931704 17800 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d: Loaded TSK: 0
I20260810 00:48:31.933239 17800 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 00:48:32.171651 17779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:48:32.171958 17779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:48:32.172127 17779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:48:32.179317 17779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:48:32.179461 17779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:48:32.179629 17779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.1
I20260810 00:48:32.189220 17779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:44267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.1:39615
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.6.1
--webserver_port=42113
--tserver_master_addrs=127.28.6.62:36385
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.1
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:48:32.190994 17779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:48:32.193209 17779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:48:32.210397 17821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:32.210373 17823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:32.211607 17830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:48:32.212672 17779 server_base.cc:1034] running on GCE node
I20260810 00:48:32.213327 17779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:48:32.214320 17779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:48:32.215550 17779 hybrid_clock.cc:648] HybridClock initialized: now 1786322912215494 us; error 48 us; skew 500 ppm
I20260810 00:48:32.215874 17779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:48:32 dist-test-slave-1nl6 krb5kdc[16802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322912, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:48:32.219245 17779 init.cc:377] Logged in from keytab as kudu/127.28.6.1@KRBTEST.COM (short username kudu)
I20260810 00:48:32.220772 17779 webserver.cc:460] Webserver started at http://127.28.6.1:42113/ using document root <none> and password file <none>
I20260810 00:48:32.221519 17779 fs_manager.cc:359] Metadata directory not provided
I20260810 00:48:32.221660 17779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:48:32.227072 17779 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 00:48:32.229820 17838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:32.231343 17779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 00:48:32.231508 17779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/wal
uuid: "5baea45d390a4f0ab315bcf66cf1687a"
format_stamp: "Formatted at 2026-08-10 00:48:29 on dist-test-slave-1nl6"
server_key: "1ef7b79bab9d263431028abc0ea015d8"
server_key_iv: "1077defa0428963b27f8169416d1e5df"
server_key_version: "encryptionkey@0"
I20260810 00:48:32.232777 17779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:48:32.253419 17779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:48:32.257763 17779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:48:32.258044 17779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:48:32.259109 17779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:48:32.260531 17779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:48:32.260653 17779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:32.260802 17779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:48:32.260893 17779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:32.293681 17779 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.1:39615
I20260810 00:48:32.293766 18011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.1:39615 every 8 connection(s)
I20260810 00:48:32.295179 17779 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
I20260810 00:48:32.301076 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 17779
I20260810 00:48:32.301726 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.2:46419
--local_ip_for_outbound_sockets=127.28.6.2
--tserver_master_addrs=127.28.6.62:36385
--webserver_port=40167
--webserver_interface=127.28.6.2
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:44267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 00:48:32 dist-test-slave-1nl6 krb5kdc[16802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322912, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:48:32.320042 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:32.297886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:35953 (local address 127.28.6.62:36385)
0810 00:48:32.298499 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:32.298511 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:32.298997 (+ 486us) server_negotiation.cc:407] Connection header received
0810 00:48:32.301275 (+ 2278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:32.301289 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:32.301591 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:32.301946 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:32.303941 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:32.305137 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:32.307692 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:32.308319 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:32.308450 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:32.312604 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:32.312612 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:32.312626 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:32.312659 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:32.315652 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:32.316598 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:32.316605 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:32.316610 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:32.316771 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:32.317271 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:32.317274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:32.317276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:32.317710 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:32.317858 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:32.318743 (+ 885us) server_negotiation.cc:299] Negotiation successful
0810 00:48:32.319116 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":266,"threads_started":1}
I20260810 00:48:32.320713 18016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:32.298008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.1:35953)
0810 00:48:32.298814 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:32.298860 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:32.300710 (+ 1850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:32.302100 (+ 1390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:32.302113 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:32.302892 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:48:32.303715 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:32.303744 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:32.305320 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:32.305328 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:32.307550 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:32.307570 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:32.308566 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:32.308576 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:32.308897 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:32.309634 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:48:32.309671 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:48:32.312310 (+ 2639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:48:32.315810 (+ 3500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:32.315821 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:32.315837 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:32.316457 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:32.316893 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:32.316901 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:32.316908 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:32.317151 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:32.317848 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:48:32.317858 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:48:32.318267 (+ 409us) client_negotiation.cc:769] Sending connection context
0810 00:48:32.318557 (+ 290us) client_negotiation.cc:240] Negotiation successful
0810 00:48:32.319152 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":341,"threads_started":1}
I20260810 00:48:32.323802 18012 heartbeater.cc:347] Connected to a master server at 127.28.6.62:36385
I20260810 00:48:32.324095 18012 heartbeater.cc:464] Registering TS with master...
I20260810 00:48:32.324841 18012 heartbeater.cc:511] Master 127.28.6.62:36385 requested a full tablet report, sending...
I20260810 00:48:32.327396 17715 ts_manager.cc:194] Registered new tserver with Master: 5baea45d390a4f0ab315bcf66cf1687a (127.28.6.1:39615)
I20260810 00:48:32.329764 17715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.1@KRBTEST.COM'} at 127.28.6.1:35953
W20260810 00:48:32.641525 18017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:48:32.641784 18017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:48:32.641873 18017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:48:32.648772 18017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:48:32.648877 18017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:48:32.648994 18017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.2
I20260810 00:48:32.658435 18017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:44267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.2:46419
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.6.2
--webserver_port=40167
--tserver_master_addrs=127.28.6.62:36385
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.2
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:48:32.659550 18017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:48:32.660887 18017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:48:32.673907 18023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:32.673950 18025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:32.673868 18029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:48:32.674656 18017 server_base.cc:1034] running on GCE node
I20260810 00:48:32.675257 18017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:48:32.676070 18017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:48:32.677299 18017 hybrid_clock.cc:648] HybridClock initialized: now 1786322912677255 us; error 55 us; skew 500 ppm
I20260810 00:48:32.677606 18017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:48:32 dist-test-slave-1nl6 krb5kdc[16802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322912, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:48:32.680660 18017 init.cc:377] Logged in from keytab as kudu/127.28.6.2@KRBTEST.COM (short username kudu)
I20260810 00:48:32.682092 18017 webserver.cc:460] Webserver started at http://127.28.6.2:40167/ using document root <none> and password file <none>
I20260810 00:48:32.682723 18017 fs_manager.cc:359] Metadata directory not provided
I20260810 00:48:32.682878 18017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:48:32.687961 18017 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 00:48:32.690498 18042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:32.691823 18017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 00:48:32.692001 18017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/wal
uuid: "d3b6361ac1dd42839442096b8b99e033"
format_stamp: "Formatted at 2026-08-10 00:48:29 on dist-test-slave-1nl6"
server_key: "086f064ec1d4b39c7d60f3b44315d9b4"
server_key_iv: "259a9c38e028019e411488e8107ac4a4"
server_key_version: "encryptionkey@0"
I20260810 00:48:32.693027 18017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:48:32.724184 18017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:48:32.728221 18017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:48:32.728531 18017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:48:32.729528 18017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:48:32.730846 18017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:48:32.731011 18017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:32.731144 18017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:48:32.731251 18017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:32.766534 18017 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.2:46419
I20260810 00:48:32.766621 18212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.2:46419 every 8 connection(s)
I20260810 00:48:32.767992 18017 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
I20260810 00:48:32.772617 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 18017
I20260810 00:48:32.773241 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.3:39599
--local_ip_for_outbound_sockets=127.28.6.3
--tserver_master_addrs=127.28.6.62:36385
--webserver_port=46817
--webserver_interface=127.28.6.3
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:44267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 00:48:32 dist-test-slave-1nl6 krb5kdc[16802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322912, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:48:32.788312 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:32.770531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:37087 (local address 127.28.6.62:36385)
0810 00:48:32.770705 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:32.770710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:32.771912 (+ 1202us) server_negotiation.cc:407] Connection header received
0810 00:48:32.773950 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:32.773955 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:32.774052 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:32.774166 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:32.775754 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:32.776435 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:32.777996 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:32.778408 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:32.778539 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:32.782202 (+ 3663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:32.782208 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:32.782211 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:32.782243 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:32.785055 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:32.785983 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:32.785987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:32.785989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:32.786109 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:32.786604 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:32.786607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:32.786609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:32.786885 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:32.787055 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:32.787903 (+ 848us) server_negotiation.cc:299] Negotiation successful
0810 00:48:32.788064 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 00:48:32.789978 18218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:32.771025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.2:37087)
0810 00:48:32.771734 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:32.771795 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:32.773648 (+ 1853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:32.774428 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:32.774436 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:32.774991 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:48:32.775567 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:32.775584 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:32.776636 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:32.776640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:32.777825 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:32.777838 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:32.778642 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:32.778649 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:32.778876 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:32.779615 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:48:32.779637 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:48:32.781996 (+ 2359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:48:32.785213 (+ 3217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:32.785232 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:32.785250 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:32.785863 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:32.786239 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:32.786246 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:32.786252 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:32.786492 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:32.787066 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:48:32.787076 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:48:32.787531 (+ 455us) client_negotiation.cc:769] Sending connection context
0810 00:48:32.787827 (+ 296us) client_negotiation.cc:240] Negotiation successful
0810 00:48:32.788427 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":269,"threads_started":1}
I20260810 00:48:32.791509 18213 heartbeater.cc:347] Connected to a master server at 127.28.6.62:36385
I20260810 00:48:32.791853 18213 heartbeater.cc:464] Registering TS with master...
I20260810 00:48:32.792760 18213 heartbeater.cc:511] Master 127.28.6.62:36385 requested a full tablet report, sending...
I20260810 00:48:32.794361 17715 ts_manager.cc:194] Registered new tserver with Master: d3b6361ac1dd42839442096b8b99e033 (127.28.6.2:46419)
I20260810 00:48:32.795265 17715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.2@KRBTEST.COM'} at 127.28.6.2:37087
W20260810 00:48:33.117949 18219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:48:33.118216 18219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:48:33.118425 18219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:48:33.125864 18219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:48:33.125964 18219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:48:33.126163 18219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.3
I20260810 00:48:33.135757 18219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:44267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.3:39599
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.6.3
--webserver_port=46817
--tserver_master_addrs=127.28.6.62:36385
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.3
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:48:33.137049 18219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:48:33.138451 18219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:48:33.152132 18229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:33.152199 18227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:33.152459 18225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:48:33.153172 18219 server_base.cc:1034] running on GCE node
I20260810 00:48:33.153924 18219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:48:33.154963 18219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:48:33.156188 18219 hybrid_clock.cc:648] HybridClock initialized: now 1786322913156133 us; error 47 us; skew 500 ppm
I20260810 00:48:33.156632 18219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:48:33 dist-test-slave-1nl6 krb5kdc[16802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322913, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:48:33.160935 18219 init.cc:377] Logged in from keytab as kudu/127.28.6.3@KRBTEST.COM (short username kudu)
I20260810 00:48:33.162463 18219 webserver.cc:460] Webserver started at http://127.28.6.3:46817/ using document root <none> and password file <none>
I20260810 00:48:33.163252 18219 fs_manager.cc:359] Metadata directory not provided
I20260810 00:48:33.163381 18219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:48:33.168792 18219 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 00:48:33.171404 18237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:33.172832 18219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 00:48:33.172991 18219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/wal
uuid: "e7878590bbf54ded8185416557742988"
format_stamp: "Formatted at 2026-08-10 00:48:30 on dist-test-slave-1nl6"
server_key: "56ea9e13eb293de681cbfcc4f50adb8a"
server_key_iv: "2ec817d455bd4df6613b559da0b18a6a"
server_key_version: "encryptionkey@0"
I20260810 00:48:33.173933 18219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:48:33.194258 18219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:48:33.198776 18219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:48:33.199064 18219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:48:33.199997 18219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:48:33.201308 18219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:48:33.201404 18219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:33.201486 18219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:48:33.201543 18219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:33.236035 18219 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.3:39599
I20260810 00:48:33.236093 18418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.3:39599 every 8 connection(s)
I20260810 00:48:33.237676 18219 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
I20260810 00:48:33.243357 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 18219
Aug 10 00:48:33 dist-test-slave-1nl6 krb5kdc[16802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322913, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:48:33.259159 18424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:33.244445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54160 (local address 127.28.6.62:36385)
0810 00:48:33.244952 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:33.244959 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:33.245283 (+ 324us) server_negotiation.cc:407] Connection header received
0810 00:48:33.245482 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:33.245487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:33.245540 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:33.245651 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:33.247237 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:33.247958 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:33.249292 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:33.249928 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:33.250078 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:33.252912 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:33.252919 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:33.252925 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:33.252970 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:33.256355 (+ 3385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:33.257070 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:33.257076 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:33.257079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:33.257199 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:33.257618 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:33.257623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:33.257627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:33.257927 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:33.258191 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:33.258511 (+ 320us) server_negotiation.cc:299] Negotiation successful
0810 00:48:33.258740 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":381,"thread_start_us":275,"threads_started":1}
I20260810 00:48:33.259390 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:33.240298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:37221 (local address 127.28.6.62:36385)
0810 00:48:33.240489 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:33.240494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:33.241676 (+ 1182us) server_negotiation.cc:407] Connection header received
0810 00:48:33.243607 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:33.243614 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:33.243708 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:33.243830 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:33.245766 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:33.246569 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:33.248222 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:33.248628 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:33.248776 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:33.252522 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:33.252527 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:33.252530 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:33.252560 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:33.256211 (+ 3651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:33.256937 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:33.256942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:33.256946 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:33.257076 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:33.257518 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:33.257524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:33.257527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:33.257852 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:33.258010 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:33.258735 (+ 725us) server_negotiation.cc:299] Negotiation successful
0810 00:48:33.258970 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":69}
I20260810 00:48:33.260751 18422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:33.240703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.3:37221)
0810 00:48:33.241473 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:33.241520 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:33.243269 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:33.244070 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:33.244083 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:33.244955 (+ 872us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:48:33.245579 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:33.245597 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:33.246768 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:33.246772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:33.248046 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:33.248060 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:33.248894 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:33.248903 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:33.249139 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:33.249812 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:48:33.249839 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:48:33.252295 (+ 2456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:48:33.256365 (+ 4070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:33.256374 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:33.256390 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:33.256804 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:33.257196 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:33.257201 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:33.257204 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:33.257393 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:33.257989 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:48:33.257997 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:48:33.258341 (+ 344us) client_negotiation.cc:769] Sending connection context
0810 00:48:33.258608 (+ 267us) client_negotiation.cc:240] Negotiation successful
0810 00:48:33.259369 (+ 761us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":296,"threads_started":1}
I20260810 00:48:33.262224 18420 heartbeater.cc:347] Connected to a master server at 127.28.6.62:36385
I20260810 00:48:33.262596 18420 heartbeater.cc:464] Registering TS with master...
I20260810 00:48:33.263453 18420 heartbeater.cc:511] Master 127.28.6.62:36385 requested a full tablet report, sending...
I20260810 00:48:33.264983 17714 ts_manager.cc:194] Registered new tserver with Master: e7878590bbf54ded8185416557742988 (127.28.6.3:39599)
I20260810 00:48:33.265909 17714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.3@KRBTEST.COM'} at 127.28.6.3:37221
I20260810 00:48:33.274078 28696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 00:48:33.280865 18424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:33.275024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54176 (local address 127.28.6.62:36385)
0810 00:48:33.275199 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:33.275210 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:33.275702 (+ 492us) server_negotiation.cc:407] Connection header received
0810 00:48:33.275866 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:33.275870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:33.275998 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:33.276116 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:48:33.276493 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:33.277253 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:33.278949 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:33.279419 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:33.279520 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:33.279975 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:48:33.280416 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:48:33.280520 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:33.280620 (+ 100us) server_negotiation.cc:299] Negotiation successful
0810 00:48:33.280682 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 00:48:33.281698 17714 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:54176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 00:48:33.288784 17714 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 00:48:33.324353 18438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:33.302281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:46419 (local address 127.0.0.1:53320)
0810 00:48:33.316989 (+ 14708us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:33.317007 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:33.317147 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:33.317655 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:33.317661 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:33.317685 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:48:33.317996 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:33.318005 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:33.319838 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:33.319844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:33.321665 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:33.321680 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:33.323477 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:33.323484 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:33.323695 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:33.323725 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 00:48:33.323824 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 00:48:33.323947 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14568,"mutex_wait_us":3,"thread_start_us":12281,"threads_started":1}
I20260810 00:48:33.324967 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:33.300974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44182 (local address 127.28.6.3:39599)
0810 00:48:33.314204 (+ 13230us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:33.314213 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:33.315424 (+ 1211us) server_negotiation.cc:407] Connection header received
0810 00:48:33.315524 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:33.315530 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:33.315701 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:33.315911 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:48:33.317039 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:33.318698 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:33.321592 (+ 2894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:33.322804 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:33.322958 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:33.323203 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:33.324532 (+ 1329us) server_negotiation.cc:299] Negotiation successful
0810 00:48:33.324693 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13031,"thread_start_us":12857,"threads_started":1}
I20260810 00:48:33.325294 18442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:33.302949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53320 (local address 127.28.6.2:46419)
0810 00:48:33.314418 (+ 11469us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:33.314424 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:33.317128 (+ 2704us) server_negotiation.cc:407] Connection header received
0810 00:48:33.317302 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:33.317312 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:33.317488 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:33.317670 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:48:33.318170 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:33.319659 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:33.321819 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:33.323243 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:33.323386 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:33.323577 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:33.324795 (+ 1218us) server_negotiation.cc:299] Negotiation successful
0810 00:48:33.324967 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11330,"thread_start_us":11196,"threads_started":1}
I20260810 00:48:33.325318 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:33.300644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:39599 (local address 127.0.0.1:44182)
0810 00:48:33.314868 (+ 14224us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:33.314977 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:33.315189 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:33.316275 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:33.316281 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:33.316312 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:48:33.316834 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:33.316849 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:33.318826 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:33.318833 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:33.321404 (+ 2571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:33.321426 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:33.323059 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:33.323065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:33.323242 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:33.323485 (+ 243us) client_negotiation.cc:769] Sending connection context
0810 00:48:33.323619 (+ 134us) client_negotiation.cc:240] Negotiation successful
0810 00:48:33.323796 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14021,"mutex_wait_us":1,"thread_start_us":14636,"threads_started":1}
I20260810 00:48:33.325937 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:33.300918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:39615 (local address 127.0.0.1:46328)
0810 00:48:33.317035 (+ 16117us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:33.317047 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:33.317245 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:33.318061 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:33.318066 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:33.318090 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:48:33.318453 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:33.318462 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:33.320347 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:33.320353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:33.322564 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:33.322580 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:33.325069 (+ 2489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:33.325077 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:33.325279 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:33.325326 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 00:48:33.325421 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 00:48:33.325545 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14917,"mutex_wait_us":20,"thread_start_us":15748,"threads_started":1}
I20260810 00:48:33.326148 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:33.301222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46328 (local address 127.28.6.1:39615)
0810 00:48:33.313462 (+ 12240us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:33.313475 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:33.317145 (+ 3670us) server_negotiation.cc:407] Connection header received
0810 00:48:33.317414 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:33.317422 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:33.317627 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:33.317808 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:48:33.318660 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:33.320236 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:33.322725 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:33.324583 (+ 1858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:33.324712 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:33.324923 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:33.325620 (+ 697us) server_negotiation.cc:299] Negotiation successful
0810 00:48:33.325803 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12006,"thread_start_us":12245,"threads_started":1}
I20260810 00:48:33.329227 18117 tablet_service.cc:1467] Processing CreateTablet for tablet 7c968565f23144889049a1b39b75c090 (DEFAULT_TABLE table=test-table [id=9951c837c6664913a2e862bc9b45f108]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:48:33.329720 17910 tablet_service.cc:1467] Processing CreateTablet for tablet 7c968565f23144889049a1b39b75c090 (DEFAULT_TABLE table=test-table [id=9951c837c6664913a2e862bc9b45f108]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:48:33.329720 18309 tablet_service.cc:1467] Processing CreateTablet for tablet 7c968565f23144889049a1b39b75c090 (DEFAULT_TABLE table=test-table [id=9951c837c6664913a2e862bc9b45f108]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:48:33.330567 18117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c968565f23144889049a1b39b75c090. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:48:33.331110 18309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c968565f23144889049a1b39b75c090. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:48:33.331147 17910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c968565f23144889049a1b39b75c090. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:48:33.333089 18012 heartbeater.cc:503] Master 127.28.6.62:36385 was elected leader, sending a full tablet report...
I20260810 00:48:33.344056 18446 tablet_bootstrap.cc:492] T 7c968565f23144889049a1b39b75c090 P d3b6361ac1dd42839442096b8b99e033: Bootstrap starting.
I20260810 00:48:33.347676 18447 tablet_bootstrap.cc:492] T 7c968565f23144889049a1b39b75c090 P e7878590bbf54ded8185416557742988: Bootstrap starting.
I20260810 00:48:33.347983 18446 tablet_bootstrap.cc:654] T 7c968565f23144889049a1b39b75c090 P d3b6361ac1dd42839442096b8b99e033: Neither blocks nor log segments found. Creating new log.
I20260810 00:48:33.349465 18446 log.cc:824] T 7c968565f23144889049a1b39b75c090 P d3b6361ac1dd42839442096b8b99e033: Log is configured to *not* fsync() on all Append() calls
I20260810 00:48:33.350767 18448 tablet_bootstrap.cc:492] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a: Bootstrap starting.
I20260810 00:48:33.352681 18447 tablet_bootstrap.cc:654] T 7c968565f23144889049a1b39b75c090 P e7878590bbf54ded8185416557742988: Neither blocks nor log segments found. Creating new log.
I20260810 00:48:33.352936 18446 tablet_bootstrap.cc:492] T 7c968565f23144889049a1b39b75c090 P d3b6361ac1dd42839442096b8b99e033: No bootstrap required, opened a new log
I20260810 00:48:33.353237 18446 ts_tablet_manager.cc:1397] T 7c968565f23144889049a1b39b75c090 P d3b6361ac1dd42839442096b8b99e033: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260810 00:48:33.353955 18447 log.cc:824] T 7c968565f23144889049a1b39b75c090 P e7878590bbf54ded8185416557742988: Log is configured to *not* fsync() on all Append() calls
I20260810 00:48:33.354279 18448 tablet_bootstrap.cc:654] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a: Neither blocks nor log segments found. Creating new log.
I20260810 00:48:33.355412 18448 log.cc:824] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a: Log is configured to *not* fsync() on all Append() calls
I20260810 00:48:33.357049 18447 tablet_bootstrap.cc:492] T 7c968565f23144889049a1b39b75c090 P e7878590bbf54ded8185416557742988: No bootstrap required, opened a new log
I20260810 00:48:33.357329 18447 ts_tablet_manager.cc:1397] T 7c968565f23144889049a1b39b75c090 P e7878590bbf54ded8185416557742988: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260810 00:48:33.358431 18448 tablet_bootstrap.cc:492] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a: No bootstrap required, opened a new log
I20260810 00:48:33.358693 18448 ts_tablet_manager.cc:1397] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 00:48:33.361907 18446 raft_consensus.cc:348] T 7c968565f23144889049a1b39b75c090 P d3b6361ac1dd42839442096b8b99e033 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7878590bbf54ded8185416557742988" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39599 } } peers { permanent_uuid: "5baea45d390a4f0ab315bcf66cf1687a" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 39615 } } peers { permanent_uuid: "d3b6361ac1dd42839442096b8b99e033" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46419 } }
I20260810 00:48:33.362311 18446 raft_consensus.cc:374] T 7c968565f23144889049a1b39b75c090 P d3b6361ac1dd42839442096b8b99e033 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:48:33.362437 18446 raft_consensus.cc:729] T 7c968565f23144889049a1b39b75c090 P d3b6361ac1dd42839442096b8b99e033 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3b6361ac1dd42839442096b8b99e033, State: Initialized, Role: FOLLOWER
I20260810 00:48:33.363059 18446 consensus_queue.cc:260] T 7c968565f23144889049a1b39b75c090 P d3b6361ac1dd42839442096b8b99e033 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7878590bbf54ded8185416557742988" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39599 } } peers { permanent_uuid: "5baea45d390a4f0ab315bcf66cf1687a" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 39615 } } peers { permanent_uuid: "d3b6361ac1dd42839442096b8b99e033" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46419 } }
I20260810 00:48:33.364171 18213 heartbeater.cc:503] Master 127.28.6.62:36385 was elected leader, sending a full tablet report...
I20260810 00:48:33.364717 18446 ts_tablet_manager.cc:1428] T 7c968565f23144889049a1b39b75c090 P d3b6361ac1dd42839442096b8b99e033: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260810 00:48:33.368374 18447 raft_consensus.cc:348] T 7c968565f23144889049a1b39b75c090 P e7878590bbf54ded8185416557742988 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7878590bbf54ded8185416557742988" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39599 } } peers { permanent_uuid: "5baea45d390a4f0ab315bcf66cf1687a" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 39615 } } peers { permanent_uuid: "d3b6361ac1dd42839442096b8b99e033" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46419 } }
I20260810 00:48:33.368709 18447 raft_consensus.cc:374] T 7c968565f23144889049a1b39b75c090 P e7878590bbf54ded8185416557742988 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:48:33.368827 18447 raft_consensus.cc:729] T 7c968565f23144889049a1b39b75c090 P e7878590bbf54ded8185416557742988 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7878590bbf54ded8185416557742988, State: Initialized, Role: FOLLOWER
I20260810 00:48:33.369396 18448 raft_consensus.cc:348] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7878590bbf54ded8185416557742988" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39599 } } peers { permanent_uuid: "5baea45d390a4f0ab315bcf66cf1687a" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 39615 } } peers { permanent_uuid: "d3b6361ac1dd42839442096b8b99e033" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46419 } }
I20260810 00:48:33.369444 18447 consensus_queue.cc:260] T 7c968565f23144889049a1b39b75c090 P e7878590bbf54ded8185416557742988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7878590bbf54ded8185416557742988" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39599 } } peers { permanent_uuid: "5baea45d390a4f0ab315bcf66cf1687a" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 39615 } } peers { permanent_uuid: "d3b6361ac1dd42839442096b8b99e033" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46419 } }
I20260810 00:48:33.369752 18448 raft_consensus.cc:374] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:48:33.369874 18448 raft_consensus.cc:729] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5baea45d390a4f0ab315bcf66cf1687a, State: Initialized, Role: FOLLOWER
I20260810 00:48:33.370451 18448 consensus_queue.cc:260] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7878590bbf54ded8185416557742988" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39599 } } peers { permanent_uuid: "5baea45d390a4f0ab315bcf66cf1687a" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 39615 } } peers { permanent_uuid: "d3b6361ac1dd42839442096b8b99e033" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46419 } }
I20260810 00:48:33.370959 18420 heartbeater.cc:503] Master 127.28.6.62:36385 was elected leader, sending a full tablet report...
I20260810 00:48:33.372166 18448 ts_tablet_manager.cc:1428] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
I20260810 00:48:33.372505 18447 ts_tablet_manager.cc:1428] T 7c968565f23144889049a1b39b75c090 P e7878590bbf54ded8185416557742988: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260810 00:48:33.411434 18454 raft_consensus.cc:482] T 7c968565f23144889049a1b39b75c090 P d3b6361ac1dd42839442096b8b99e033 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:48:33.411736 18454 raft_consensus.cc:504] T 7c968565f23144889049a1b39b75c090 P d3b6361ac1dd42839442096b8b99e033 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7878590bbf54ded8185416557742988" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39599 } } peers { permanent_uuid: "5baea45d390a4f0ab315bcf66cf1687a" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 39615 } } peers { permanent_uuid: "d3b6361ac1dd42839442096b8b99e033" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46419 } }
I20260810 00:48:33.413465 18454 leader_election.cc:288] T 7c968565f23144889049a1b39b75c090 P d3b6361ac1dd42839442096b8b99e033 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7878590bbf54ded8185416557742988 (127.28.6.3:39599), 5baea45d390a4f0ab315bcf66cf1687a (127.28.6.1:39615)
I20260810 00:48:33.420432 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:33.414102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:59539 (local address 127.28.6.1:39615)
0810 00:48:33.414330 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:33.414339 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:33.415016 (+ 677us) server_negotiation.cc:407] Connection header received
0810 00:48:33.415229 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:33.415241 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:33.415303 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:33.415398 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:48:33.415992 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:33.416762 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:33.418662 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:33.419487 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:33.419638 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:33.419761 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:33.420051 (+ 290us) server_negotiation.cc:299] Negotiation successful
0810 00:48:33.420177 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 00:48:33.420497 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:33.413840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:55115 (local address 127.28.6.3:39599)
0810 00:48:33.414098 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:33.414108 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:33.415194 (+ 1086us) server_negotiation.cc:407] Connection header received
0810 00:48:33.415394 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:33.415398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:33.415455 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:33.415552 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:48:33.416194 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:33.416913 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:33.418720 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:33.419558 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:33.419700 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:33.419808 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:33.420115 (+ 307us) server_negotiation.cc:299] Negotiation successful
0810 00:48:33.420245 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260810 00:48:33.420640 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:33.413800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:39599 (local address 127.28.6.2:55115)
0810 00:48:33.415087 (+ 1287us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:33.415101 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:33.415244 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:33.415569 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:33.415574 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:33.415626 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:48:33.416041 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:33.416049 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:33.417053 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:33.417061 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:33.418563 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:33.418575 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:33.419799 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:33.419806 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:33.419960 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:33.419983 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 00:48:33.420088 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 00:48:33.420211 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1136,"mutex_wait_us":4,"spinlock_wait_cycles":1664,"thread_start_us":536,"threads_started":1}
I20260810 00:48:33.420990 18461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:33.413794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:39615 (local address 127.28.6.2:59539)
0810 00:48:33.414863 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:33.414879 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:33.415086 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:33.415414 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:33.415419 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:33.415442 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:48:33.415837 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:33.415846 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:33.416893 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:33.416899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:33.418505 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:33.418520 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:33.419741 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:33.419747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:33.419888 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:33.419916 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 00:48:33.420023 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 00:48:33.420153 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":951,"mutex_wait_us":25,"spinlock_wait_cycles":24320,"thread_start_us":914,"threads_started":1}
I20260810 00:48:33.421739 18340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c968565f23144889049a1b39b75c090" candidate_uuid: "d3b6361ac1dd42839442096b8b99e033" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7878590bbf54ded8185416557742988" is_pre_election: true
I20260810 00:48:33.422086 17944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c968565f23144889049a1b39b75c090" candidate_uuid: "d3b6361ac1dd42839442096b8b99e033" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5baea45d390a4f0ab315bcf66cf1687a" is_pre_election: true
I20260810 00:48:33.422425 18340 raft_consensus.cc:2454] T 7c968565f23144889049a1b39b75c090 P e7878590bbf54ded8185416557742988 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3b6361ac1dd42839442096b8b99e033 in term 0.
I20260810 00:48:33.422633 17944 raft_consensus.cc:2454] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3b6361ac1dd42839442096b8b99e033 in term 0.
I20260810 00:48:33.423157 18046 leader_election.cc:302] T 7c968565f23144889049a1b39b75c090 P d3b6361ac1dd42839442096b8b99e033 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3b6361ac1dd42839442096b8b99e033, e7878590bbf54ded8185416557742988; no voters:
I20260810 00:48:33.423530 18454 raft_consensus.cc:2781] T 7c968565f23144889049a1b39b75c090 P d3b6361ac1dd42839442096b8b99e033 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 00:48:33.423677 18454 raft_consensus.cc:482] T 7c968565f23144889049a1b39b75c090 P d3b6361ac1dd42839442096b8b99e033 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 00:48:33.423792 18454 raft_consensus.cc:3037] T 7c968565f23144889049a1b39b75c090 P d3b6361ac1dd42839442096b8b99e033 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:48:33.425981 18454 raft_consensus.cc:504] T 7c968565f23144889049a1b39b75c090 P d3b6361ac1dd42839442096b8b99e033 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7878590bbf54ded8185416557742988" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39599 } } peers { permanent_uuid: "5baea45d390a4f0ab315bcf66cf1687a" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 39615 } } peers { permanent_uuid: "d3b6361ac1dd42839442096b8b99e033" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46419 } }
I20260810 00:48:33.426620 18454 leader_election.cc:288] T 7c968565f23144889049a1b39b75c090 P d3b6361ac1dd42839442096b8b99e033 [CANDIDATE]: Term 1 election: Requested vote from peers e7878590bbf54ded8185416557742988 (127.28.6.3:39599), 5baea45d390a4f0ab315bcf66cf1687a (127.28.6.1:39615)
I20260810 00:48:33.427032 18340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c968565f23144889049a1b39b75c090" candidate_uuid: "d3b6361ac1dd42839442096b8b99e033" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7878590bbf54ded8185416557742988"
I20260810 00:48:33.427167 17944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c968565f23144889049a1b39b75c090" candidate_uuid: "d3b6361ac1dd42839442096b8b99e033" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5baea45d390a4f0ab315bcf66cf1687a"
I20260810 00:48:33.427232 18340 raft_consensus.cc:3037] T 7c968565f23144889049a1b39b75c090 P e7878590bbf54ded8185416557742988 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:48:33.427380 17944 raft_consensus.cc:3037] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:48:33.429461 18340 raft_consensus.cc:2454] T 7c968565f23144889049a1b39b75c090 P e7878590bbf54ded8185416557742988 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3b6361ac1dd42839442096b8b99e033 in term 1.
I20260810 00:48:33.429606 17944 raft_consensus.cc:2454] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3b6361ac1dd42839442096b8b99e033 in term 1.
I20260810 00:48:33.429941 18046 leader_election.cc:302] T 7c968565f23144889049a1b39b75c090 P d3b6361ac1dd42839442096b8b99e033 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3b6361ac1dd42839442096b8b99e033, e7878590bbf54ded8185416557742988; no voters:
I20260810 00:48:33.430246 18454 raft_consensus.cc:2781] T 7c968565f23144889049a1b39b75c090 P d3b6361ac1dd42839442096b8b99e033 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:48:33.430644 18454 raft_consensus.cc:686] T 7c968565f23144889049a1b39b75c090 P d3b6361ac1dd42839442096b8b99e033 [term 1 LEADER]: Becoming Leader. State: Replica: d3b6361ac1dd42839442096b8b99e033, State: Running, Role: LEADER
I20260810 00:48:33.431174 18454 consensus_queue.cc:237] T 7c968565f23144889049a1b39b75c090 P d3b6361ac1dd42839442096b8b99e033 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7878590bbf54ded8185416557742988" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39599 } } peers { permanent_uuid: "5baea45d390a4f0ab315bcf66cf1687a" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 39615 } } peers { permanent_uuid: "d3b6361ac1dd42839442096b8b99e033" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46419 } }
I20260810 00:48:33.435916 17714 catalog_manager.cc:5495] T 7c968565f23144889049a1b39b75c090 P d3b6361ac1dd42839442096b8b99e033 reported cstate change: term changed from 0 to 1, leader changed from <none> to d3b6361ac1dd42839442096b8b99e033 (127.28.6.2). New cstate: current_term: 1 leader_uuid: "d3b6361ac1dd42839442096b8b99e033" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7878590bbf54ded8185416557742988" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5baea45d390a4f0ab315bcf66cf1687a" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 39615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3b6361ac1dd42839442096b8b99e033" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46419 } health_report { overall_health: HEALTHY } } }
I20260810 00:48:33.476019 18442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:33.470416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53328 (local address 127.28.6.2:46419)
0810 00:48:33.470625 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:33.470631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:33.470650 (+ 19us) server_negotiation.cc:407] Connection header received
0810 00:48:33.470843 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:33.470847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:33.471014 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:33.471166 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:48:33.471546 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:33.472295 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:33.473668 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:33.474075 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:33.474214 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:33.474723 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:48:33.475265 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:48:33.475552 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:33.475643 (+ 91us) server_negotiation.cc:299] Negotiation successful
0810 00:48:33.475741 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 00:48:33.484354 18340 raft_consensus.cc:1261] T 7c968565f23144889049a1b39b75c090 P e7878590bbf54ded8185416557742988 [term 1 FOLLOWER]: Refusing update from remote peer d3b6361ac1dd42839442096b8b99e033: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:48:33.484365 17944 raft_consensus.cc:1261] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a [term 1 FOLLOWER]: Refusing update from remote peer d3b6361ac1dd42839442096b8b99e033: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:48:33.485332 18462 consensus_queue.cc:1035] T 7c968565f23144889049a1b39b75c090 P d3b6361ac1dd42839442096b8b99e033 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7878590bbf54ded8185416557742988" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:48:33.485678 18454 consensus_queue.cc:1035] T 7c968565f23144889049a1b39b75c090 P d3b6361ac1dd42839442096b8b99e033 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5baea45d390a4f0ab315bcf66cf1687a" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 39615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 00:48:33.489323 18421 tablet.cc:2369] T 7c968565f23144889049a1b39b75c090 P e7878590bbf54ded8185416557742988: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:48:33.511572 17714 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:54176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:48:33.511849 17714 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:54176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:48:33.515151 17714 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d: Sending DeleteTablet for 3 replicas of tablet 7c968565f23144889049a1b39b75c090
I20260810 00:48:33.516316 18309 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c968565f23144889049a1b39b75c090 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:48:33 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:44182
I20260810 00:48:33.516431 17910 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c968565f23144889049a1b39b75c090 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:48:33 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:46328
I20260810 00:48:33.516628 18117 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c968565f23144889049a1b39b75c090 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:48:33 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:53320
I20260810 00:48:33.517324 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 17779
I20260810 00:48:33.552196 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:33.518449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46338 (local address 127.28.6.1:39615)
0810 00:48:33.518619 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:33.518625 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:33.518645 (+ 20us) server_negotiation.cc:407] Connection header received
0810 00:48:33.518797 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:33.518807 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:33.518961 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:33.519081 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:33.523106 (+ 4025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:33.524618 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:33.528881 (+ 4263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:33.529788 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:33.529910 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:33.539577 (+ 9667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:33.539598 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:33.539624 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:33.539687 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:33.544906 (+ 5219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:33.546661 (+ 1755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:33.546675 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:33.546681 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:33.546881 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:33.548280 (+ 1399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:33.548288 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:33.548294 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:33.548893 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:33.549289 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:33.551335 (+ 2046us) server_negotiation.cc:299] Negotiation successful
0810 00:48:33.551639 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 00:48:33.553047 18006 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:48:33.560520 18476 tablet_replica.cc:330] T 7c968565f23144889049a1b39b75c090 P e7878590bbf54ded8185416557742988: stopping tablet replica
I20260810 00:48:33.561120 18476 raft_consensus.cc:2229] T 7c968565f23144889049a1b39b75c090 P e7878590bbf54ded8185416557742988 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:48:33.561790 18476 raft_consensus.cc:2258] T 7c968565f23144889049a1b39b75c090 P e7878590bbf54ded8185416557742988 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:48:33.562772 18478 tablet_replica.cc:330] T 7c968565f23144889049a1b39b75c090 P d3b6361ac1dd42839442096b8b99e033: stopping tablet replica
I20260810 00:48:33.564076 18478 raft_consensus.cc:2229] T 7c968565f23144889049a1b39b75c090 P d3b6361ac1dd42839442096b8b99e033 [term 1 LEADER]: Raft consensus shutting down.
I20260810 00:48:33.564926 18478 raft_consensus.cc:2258] T 7c968565f23144889049a1b39b75c090 P d3b6361ac1dd42839442096b8b99e033 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:48:33.567353 18478 ts_tablet_manager.cc:1905] T 7c968565f23144889049a1b39b75c090 P d3b6361ac1dd42839442096b8b99e033: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:48:33.568512 18476 ts_tablet_manager.cc:1905] T 7c968565f23144889049a1b39b75c090 P e7878590bbf54ded8185416557742988: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:48:33.575192 18478 ts_tablet_manager.cc:1918] T 7c968565f23144889049a1b39b75c090 P d3b6361ac1dd42839442096b8b99e033: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:48:33.575482 18478 log.cc:1196] T 7c968565f23144889049a1b39b75c090 P d3b6361ac1dd42839442096b8b99e033: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/wal/wals/7c968565f23144889049a1b39b75c090
I20260810 00:48:33.576200 18478 ts_tablet_manager.cc:1939] T 7c968565f23144889049a1b39b75c090 P d3b6361ac1dd42839442096b8b99e033: Deleting consensus metadata
I20260810 00:48:33.576354 18476 ts_tablet_manager.cc:1918] T 7c968565f23144889049a1b39b75c090 P e7878590bbf54ded8185416557742988: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:48:33.576521 18476 log.cc:1196] T 7c968565f23144889049a1b39b75c090 P e7878590bbf54ded8185416557742988: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/wal/wals/7c968565f23144889049a1b39b75c090
I20260810 00:48:33.577291 18476 ts_tablet_manager.cc:1939] T 7c968565f23144889049a1b39b75c090 P e7878590bbf54ded8185416557742988: Deleting consensus metadata
I20260810 00:48:33.578502 17689 catalog_manager.cc:4841] TS d3b6361ac1dd42839442096b8b99e033 (127.28.6.2:46419): tablet 7c968565f23144889049a1b39b75c090 (table test-table [id=9951c837c6664913a2e862bc9b45f108]) successfully deleted
I20260810 00:48:33.579173 17694 catalog_manager.cc:4841] TS e7878590bbf54ded8185416557742988 (127.28.6.3:39599): tablet 7c968565f23144889049a1b39b75c090 (table test-table [id=9951c837c6664913a2e862bc9b45f108]) successfully deleted
I20260810 00:48:33.659441 18477 tablet_replica.cc:330] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a: stopping tablet replica
I20260810 00:48:33.659905 18477 raft_consensus.cc:2229] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:48:33.660398 18477 raft_consensus.cc:2258] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:48:33.661136 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 17779
W20260810 00:48:33.687114 17695 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.6.1:39615 (error 108)
I20260810 00:48:33.687397 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 18017
I20260810 00:48:33.689237 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:33.688704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:39615 (local address 127.0.0.1:46342)
0810 00:48:33.688932 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:33.689059 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.1:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260810 00:48:33.689499 17695 catalog_manager.cc:4584] TS 5baea45d390a4f0ab315bcf66cf1687a (127.28.6.1:39615): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7c968565f23144889049a1b39b75c090: Network error: Client connection negotiation failed: client connection to 127.28.6.1:39615: connect: Connection refused (error 111)
I20260810 00:48:33.703788 18442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:33.688232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53344 (local address 127.28.6.2:46419)
0810 00:48:33.688452 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:33.688458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:33.688591 (+ 133us) server_negotiation.cc:407] Connection header received
0810 00:48:33.688817 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:33.688823 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:33.688920 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:33.689046 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:33.690960 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:33.691883 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:33.693836 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:33.694289 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:33.694381 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:33.697803 (+ 3422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:33.697813 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:33.697829 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:33.697874 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:33.700996 (+ 3122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:33.701769 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:33.701776 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:33.701778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:33.701908 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:33.702303 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:33.702311 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:33.702313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:33.702681 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:33.702979 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:33.703294 (+ 315us) server_negotiation.cc:299] Negotiation successful
0810 00:48:33.703451 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260810 00:48:33.704495 18206 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:48:33.743144 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:33.742561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:39615 (local address 127.0.0.1:46348)
0810 00:48:33.742799 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:33.742939 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.1:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260810 00:48:33.744267 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:33.743882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:39615 (local address 127.0.0.1:46364)
0810 00:48:33.744033 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:33.744128 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.1:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 00:48:33.804925 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:33.804372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:39615 (local address 127.0.0.1:46380)
0810 00:48:33.804595 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:33.804709 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.1:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260810 00:48:33.806231 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:33.805765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:39615 (local address 127.0.0.1:46392)
0810 00:48:33.806005 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:33.806085 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.1:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260810 00:48:33.827474 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 18017
I20260810 00:48:33.855911 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 18219
I20260810 00:48:33.869877 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:33.856691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44190 (local address 127.28.6.3:39599)
0810 00:48:33.856884 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:33.856890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:33.856966 (+ 76us) server_negotiation.cc:407] Connection header received
0810 00:48:33.857183 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:33.857189 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:33.857270 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:33.857382 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:33.858729 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:33.859541 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:33.861018 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:33.861494 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:33.861631 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:33.864308 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:33.864325 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:33.864340 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:33.864375 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:33.867088 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:33.867743 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:33.867753 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:33.867756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:33.867920 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:33.868299 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:33.868304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:33.868308 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:33.868660 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:33.868974 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:33.869330 (+ 356us) server_negotiation.cc:299] Negotiation successful
0810 00:48:33.869569 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 00:48:33.870533 18409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:48:33.886971 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:33.886459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:39615 (local address 127.0.0.1:46404)
0810 00:48:33.886676 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:33.886781 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.1:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260810 00:48:33.888094 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:33.887635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:39615 (local address 127.0.0.1:46418)
0810 00:48:33.887862 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:33.887959 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.1:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260810 00:48:33.959914 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 18219
I20260810 00:48:33.986792 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 17662
I20260810 00:48:33.988091 17767 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:48:34.102082 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:34.101402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:39615 (local address 127.0.0.1:46420)
0810 00:48:34.101739 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:34.101849 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.1:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260810 00:48:34.102464 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 17662
I20260810 00:48:34.133354 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:36385
--webserver_interface=127.28.6.62
--webserver_port=43747
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:44267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:36385
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:48:34.475574 18488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:48:34.475837 18488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:48:34.475940 18488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:48:34.482933 18488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:48:34.483034 18488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:48:34.483084 18488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:48:34.483151 18488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:48:34.483196 18488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:48:34.492615 18488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:44267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:36385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:36385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=43747
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:48:34.493806 18488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:48:34.495079 18488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:48:34.506716 18500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:34.506807 18499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:34.506758 18502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:48:34.507689 18488 server_base.cc:1034] running on GCE node
I20260810 00:48:34.508404 18488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:48:34.509646 18488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:48:34.510859 18488 hybrid_clock.cc:648] HybridClock initialized: now 1786322914510796 us; error 54 us; skew 500 ppm
I20260810 00:48:34.511230 18488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:48:34 dist-test-slave-1nl6 krb5kdc[16802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322914, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:48:34.514237 18488 init.cc:377] Logged in from keytab as kudu/127.28.6.62@KRBTEST.COM (short username kudu)
I20260810 00:48:34.515751 18488 webserver.cc:460] Webserver started at http://127.28.6.62:43747/ using document root <none> and password file <none>
I20260810 00:48:34.516409 18488 fs_manager.cc:359] Metadata directory not provided
I20260810 00:48:34.516558 18488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:48:34.521893 18488 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 00:48:34.524674 18512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:34.526000 18488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 00:48:34.526180 18488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "4fd535fd266f44949f3ea71c1aefee5d"
format_stamp: "Formatted at 2026-08-10 00:48:28 on dist-test-slave-1nl6"
server_key: "2171d19594674b98514a47a2cade09e3"
server_key_iv: "50b9e3e15cbf1da654407d5cac7b55d8"
server_key_version: "encryptionkey@0"
I20260810 00:48:34.527186 18488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:48:34.539445 18488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:48:34.543164 18488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:48:34.543432 18488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:48:34.561829 18488 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:36385
I20260810 00:48:34.561854 18588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:36385 every 8 connection(s)
I20260810 00:48:34.563341 18488 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:48:34.569818 18589 sys_catalog.cc:263] Verifying existing consensus state
I20260810 00:48:34.572059 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 18488
I20260810 00:48:34.572919 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.1:39615
--local_ip_for_outbound_sockets=127.28.6.1
--tserver_master_addrs=127.28.6.62:36385
--webserver_port=42113
--webserver_interface=127.28.6.1
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:44267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 00:48:34.573148 18589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d: Bootstrap starting.
I20260810 00:48:34.593173 18589 log.cc:824] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d: Log is configured to *not* fsync() on all Append() calls
I20260810 00:48:34.609215 18589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 00:48:34.609756 18589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d: Bootstrap complete.
I20260810 00:48:34.618815 18589 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd535fd266f44949f3ea71c1aefee5d" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36385 } }
I20260810 00:48:34.619762 18589 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd535fd266f44949f3ea71c1aefee5d, State: Initialized, Role: FOLLOWER
I20260810 00:48:34.620404 18589 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd535fd266f44949f3ea71c1aefee5d" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36385 } }
I20260810 00:48:34.620659 18589 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:48:34.620792 18589 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:48:34.620987 18589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [term 2 FOLLOWER]: Advancing to term 3
I20260810 00:48:34.625115 18589 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd535fd266f44949f3ea71c1aefee5d" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36385 } }
I20260810 00:48:34.625603 18589 leader_election.cc:302] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fd535fd266f44949f3ea71c1aefee5d; no voters:
I20260810 00:48:34.626228 18589 leader_election.cc:288] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 00:48:34.626469 18596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [term 3 FOLLOWER]: Leader election won for term 3
I20260810 00:48:34.627539 18596 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [term 3 LEADER]: Becoming Leader. State: Replica: 4fd535fd266f44949f3ea71c1aefee5d, State: Running, Role: LEADER
I20260810 00:48:34.628400 18589 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:48:34.628182 18596 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd535fd266f44949f3ea71c1aefee5d" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36385 } }
I20260810 00:48:34.631187 18598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fd535fd266f44949f3ea71c1aefee5d. Latest consensus state: current_term: 3 leader_uuid: "4fd535fd266f44949f3ea71c1aefee5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd535fd266f44949f3ea71c1aefee5d" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36385 } } }
I20260810 00:48:34.631561 18598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [sys.catalog]: This master's current role is: LEADER
I20260810 00:48:34.632135 18597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "4fd535fd266f44949f3ea71c1aefee5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd535fd266f44949f3ea71c1aefee5d" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36385 } } }
I20260810 00:48:34.632424 18597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fd535fd266f44949f3ea71c1aefee5d [sys.catalog]: This master's current role is: LEADER
I20260810 00:48:34.637101 18605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:48:34.645998 18605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:48:34.647481 18605 catalog_manager.cc:1259] Loaded cluster ID: 5bee55c23a1e4e328058c3e0186cb203
I20260810 00:48:34.647588 18605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260810 00:48:34.649639 18605 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f015fb529ad google::LogMessage::Fail() at ??:0
@ 0x7f015fb5588a google::LogMessage::SendToLog() at ??:0
@ 0x7f015fb524de google::LogMessage::Flush() at ??:0
@ 0x7f015fb53a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0175585512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f0175582976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f017560a610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f017560a439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f016279e345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f01627a191c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f01627a1799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f016277ceaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0165a566db start_thread at ??:0
@ 0x7f015e9ec61f clone at ??:0
W20260810 00:48:34.994724 18592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:48:34.995010 18592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:48:34.995205 18592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:48:35.002488 18592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:48:35.002631 18592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:48:35.002825 18592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.1
I20260810 00:48:35.012591 18592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:44267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.1:39615
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.6.1
--webserver_port=42113
--tserver_master_addrs=127.28.6.62:36385
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.1
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:48:35.013808 18592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:48:35.015378 18592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:48:35.028645 18663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:35.028569 18661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:35.028605 18668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:48:35.029526 18592 server_base.cc:1034] running on GCE node
I20260810 00:48:35.030133 18592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:48:35.031105 18592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:48:35.032312 18592 hybrid_clock.cc:648] HybridClock initialized: now 1786322915032246 us; error 60 us; skew 500 ppm
I20260810 00:48:35.032586 18592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:48:35 dist-test-slave-1nl6 krb5kdc[16802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322915, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:48:35.041642 18592 init.cc:377] Logged in from keytab as kudu/127.28.6.1@KRBTEST.COM (short username kudu)
I20260810 00:48:35.043349 18592 webserver.cc:460] Webserver started at http://127.28.6.1:42113/ using document root <none> and password file <none>
I20260810 00:48:35.044059 18592 fs_manager.cc:359] Metadata directory not provided
I20260810 00:48:35.044214 18592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:48:35.049547 18592 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260810 00:48:35.052500 18681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:35.053824 18592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 00:48:35.054005 18592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/wal
uuid: "5baea45d390a4f0ab315bcf66cf1687a"
format_stamp: "Formatted at 2026-08-10 00:48:29 on dist-test-slave-1nl6"
server_key: "1ef7b79bab9d263431028abc0ea015d8"
server_key_iv: "1077defa0428963b27f8169416d1e5df"
server_key_version: "encryptionkey@0"
I20260810 00:48:35.055006 18592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:48:35.088172 18592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:48:35.092407 18592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:48:35.092644 18592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:48:35.093627 18592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:48:35.095597 18689 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260810 00:48:35.101177 18592 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 00:48:35.101348 18592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260810 00:48:35.101481 18592 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 00:48:35.104745 18592 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 00:48:35.104852 18592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.002s
I20260810 00:48:35.105309 18689 tablet_bootstrap.cc:492] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a: Bootstrap starting.
I20260810 00:48:35.132777 18689 log.cc:824] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a: Log is configured to *not* fsync() on all Append() calls
I20260810 00:48:35.151829 18592 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.1:39615
I20260810 00:48:35.151854 18858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.1:39615 every 8 connection(s)
I20260810 00:48:35.152024 18689 tablet_bootstrap.cc:492] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 00:48:35.152580 18689 tablet_bootstrap.cc:492] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a: Bootstrap complete.
I20260810 00:48:35.153234 18592 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
I20260810 00:48:35.154177 18689 ts_tablet_manager.cc:1397] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a: Time spent bootstrapping tablet: real 0.049s user 0.025s sys 0.020s
I20260810 00:48:35.158603 18861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:35.156278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.1:42059)
0810 00:48:35.157297 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:35.157602 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":718,"thread_start_us":510,"threads_started":1}
W20260810 00:48:35.159353 18859 heartbeater.cc:644] Failed to heartbeat to 127.28.6.62:36385 (0 consecutive failures): Network error: Failed to ping master at 127.28.6.62:36385: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
I20260810 00:48:35.160257 18861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:35.159897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.1:51175)
0810 00:48:35.160052 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:35.160131 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 00:48:35.161172 18861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:35.160811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.1:52461)
0810 00:48:35.160953 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:35.161035 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 00:48:35.161422 18859 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 00:48:35.162773 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 18592
I20260810 00:48:35.163653 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.2:46419
--local_ip_for_outbound_sockets=127.28.6.2
--tserver_master_addrs=127.28.6.62:36385
--webserver_port=40167
--webserver_interface=127.28.6.2
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:44267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 00:48:35.162516 18689 raft_consensus.cc:348] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7878590bbf54ded8185416557742988" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39599 } } peers { permanent_uuid: "5baea45d390a4f0ab315bcf66cf1687a" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 39615 } } peers { permanent_uuid: "d3b6361ac1dd42839442096b8b99e033" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46419 } }
I20260810 00:48:35.163832 18689 raft_consensus.cc:729] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5baea45d390a4f0ab315bcf66cf1687a, State: Initialized, Role: FOLLOWER
I20260810 00:48:35.164520 18689 consensus_queue.cc:260] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7878590bbf54ded8185416557742988" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39599 } } peers { permanent_uuid: "5baea45d390a4f0ab315bcf66cf1687a" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 39615 } } peers { permanent_uuid: "d3b6361ac1dd42839442096b8b99e033" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46419 } }
I20260810 00:48:35.166252 18689 ts_tablet_manager.cc:1428] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260810 00:48:35.167141 18861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:35.166240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.1:37865)
0810 00:48:35.166869 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:35.166984 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446}
I20260810 00:48:35.168030 18861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:35.167701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.1:47431)
0810 00:48:35.167842 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:35.167927 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 00:48:35.501008 18864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:48:35.501286 18864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:48:35.501451 18864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:48:35.508455 18864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:48:35.508584 18864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:48:35.508780 18864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.2
I20260810 00:48:35.518289 18864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:44267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.2:46419
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.6.2
--webserver_port=40167
--tserver_master_addrs=127.28.6.62:36385
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.2
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:48:35.519611 18864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:48:35.521023 18864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:48:35.534256 18872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:35.534210 18875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:35.534165 18873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:48:35.535269 18864 server_base.cc:1034] running on GCE node
I20260810 00:48:35.535849 18864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:48:35.536751 18864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:48:35.537977 18864 hybrid_clock.cc:648] HybridClock initialized: now 1786322915537922 us; error 45 us; skew 500 ppm
I20260810 00:48:35.538286 18864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:48:35 dist-test-slave-1nl6 krb5kdc[16802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322915, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:48:35.541558 18864 init.cc:377] Logged in from keytab as kudu/127.28.6.2@KRBTEST.COM (short username kudu)
I20260810 00:48:35.543107 18864 webserver.cc:460] Webserver started at http://127.28.6.2:40167/ using document root <none> and password file <none>
I20260810 00:48:35.543771 18864 fs_manager.cc:359] Metadata directory not provided
I20260810 00:48:35.543922 18864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:48:35.549095 18864 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260810 00:48:35.551846 18889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:35.553150 18864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 00:48:35.553341 18864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/wal
uuid: "d3b6361ac1dd42839442096b8b99e033"
format_stamp: "Formatted at 2026-08-10 00:48:29 on dist-test-slave-1nl6"
server_key: "086f064ec1d4b39c7d60f3b44315d9b4"
server_key_iv: "259a9c38e028019e411488e8107ac4a4"
server_key_version: "encryptionkey@0"
I20260810 00:48:35.554302 18864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:48:35.575452 18864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:48:35.579510 18864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:48:35.579792 18864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:48:35.580785 18864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:48:35.582118 18864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:48:35.582247 18864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:35.582387 18864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:48:35.582484 18864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:35.615660 18864 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.2:46419
I20260810 00:48:35.615757 19074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.2:46419 every 8 connection(s)
I20260810 00:48:35.617122 18864 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
I20260810 00:48:35.621552 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 18864
I20260810 00:48:35.621937 19079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:35.620032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.2:35937)
0810 00:48:35.620770 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:35.620952 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":272,"threads_started":1}
I20260810 00:48:35.622265 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.3:39599
--local_ip_for_outbound_sockets=127.28.6.3
--tserver_master_addrs=127.28.6.62:36385
--webserver_port=46817
--webserver_interface=127.28.6.3
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:44267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:48:35.622588 19077 heartbeater.cc:644] Failed to heartbeat to 127.28.6.62:36385 (0 consecutive failures): Network error: Failed to ping master at 127.28.6.62:36385: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
I20260810 00:48:35.623409 19079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:35.623086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.2:37699)
0810 00:48:35.623232 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:35.623308 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 00:48:35.624359 19079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:35.623936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.2:42057)
0810 00:48:35.624107 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:35.624205 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260810 00:48:35.624604 19077 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 00:48:35.949174 19080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:48:35.949513 19080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:48:35.949695 19080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:48:35.956751 19080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:48:35.956895 19080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:48:35.957072 19080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.3
I20260810 00:48:35.966706 19080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:44267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.3:39599
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.6.3
--webserver_port=46817
--tserver_master_addrs=127.28.6.62:36385
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.3
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:48:35.967967 19080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:48:35.969281 19080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:48:35.982169 19091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:35.982306 19089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:35.982249 19095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:48:35.983187 19080 server_base.cc:1034] running on GCE node
I20260810 00:48:35.983711 19080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:48:35.984594 19080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:48:35.985811 19080 hybrid_clock.cc:648] HybridClock initialized: now 1786322915985752 us; error 53 us; skew 500 ppm
I20260810 00:48:35.986130 19080 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:48:35 dist-test-slave-1nl6 krb5kdc[16802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322915, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:48:35.989209 19080 init.cc:377] Logged in from keytab as kudu/127.28.6.3@KRBTEST.COM (short username kudu)
I20260810 00:48:35.990550 19080 webserver.cc:460] Webserver started at http://127.28.6.3:46817/ using document root <none> and password file <none>
I20260810 00:48:35.991276 19080 fs_manager.cc:359] Metadata directory not provided
I20260810 00:48:35.991411 19080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:48:35.996414 19080 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260810 00:48:35.999025 19106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:36.000313 19080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 00:48:36.000492 19080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/wal
uuid: "e7878590bbf54ded8185416557742988"
format_stamp: "Formatted at 2026-08-10 00:48:30 on dist-test-slave-1nl6"
server_key: "56ea9e13eb293de681cbfcc4f50adb8a"
server_key_iv: "2ec817d455bd4df6613b559da0b18a6a"
server_key_version: "encryptionkey@0"
I20260810 00:48:36.001497 19080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:48:36.025416 19080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:48:36.029184 19080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:48:36.029446 19080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:48:36.030413 19080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:48:36.031739 19080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:48:36.031873 19080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:36.032004 19080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:48:36.032124 19080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:36.066604 19080 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.3:39599
I20260810 00:48:36.066695 19293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.3:39599 every 8 connection(s)
I20260810 00:48:36.068101 19080 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
I20260810 00:48:36.068660 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 19080
I20260810 00:48:36.073101 19299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:36.071279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.3:39527)
0810 00:48:36.072011 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:36.072262 (+ 251us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":276,"threads_started":1}
W20260810 00:48:36.073836 19294 heartbeater.cc:644] Failed to heartbeat to 127.28.6.62:36385 (0 consecutive failures): Network error: Failed to ping master at 127.28.6.62:36385: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
I20260810 00:48:36.074564 19299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:36.074277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.3:54343)
0810 00:48:36.074404 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:36.074468 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 00:48:36.075488 19299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:36.075151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.3:49383)
0810 00:48:36.075310 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:36.075380 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 00:48:36.075783 19294 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 00:48:36.170454 19302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:36.168941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.1:33363)
0810 00:48:36.170112 (+ 1171us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:36.170254 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1046,"thread_start_us":894,"threads_started":1}
I20260810 00:48:36.625944 19307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:36.625067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.2:57215)
0810 00:48:36.625566 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:36.625684 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":230,"threads_started":1}
I20260810 00:48:36.729427 19308 raft_consensus.cc:482] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:48:36.729758 19308 raft_consensus.cc:504] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7878590bbf54ded8185416557742988" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39599 } } peers { permanent_uuid: "5baea45d390a4f0ab315bcf66cf1687a" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 39615 } } peers { permanent_uuid: "d3b6361ac1dd42839442096b8b99e033" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46419 } }
I20260810 00:48:36.731523 19308 leader_election.cc:288] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e7878590bbf54ded8185416557742988 (127.28.6.3:39599), d3b6361ac1dd42839442096b8b99e033 (127.28.6.2:46419)
Aug 10 00:48:36 dist-test-slave-1nl6 krb5kdc[16802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322915, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for kudu/127.28.6.2@KRBTEST.COM
Aug 10 00:48:36 dist-test-slave-1nl6 krb5kdc[16802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322915, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for kudu/127.28.6.3@KRBTEST.COM
I20260810 00:48:36.770977 19311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:36.731742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:46419 (local address 127.28.6.1:52265)
0810 00:48:36.745899 (+ 14157us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:36.745963 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:36.746732 (+ 769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:36.750410 (+ 3678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:36.750456 (+ 46us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:36.751745 (+ 1289us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:48:36.752990 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:36.753027 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:36.755379 (+ 2352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:36.755383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:36.757861 (+ 2478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:36.757878 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:36.759624 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:36.759631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:36.759934 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:36.760892 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:48:36.760929 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:48:36.763573 (+ 2644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:48:36.767273 (+ 3700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:36.767282 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:36.767296 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:36.767809 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:36.768320 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:36.768328 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:36.768332 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:36.768540 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:36.769338 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:48:36.769343 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:48:36.769638 (+ 295us) client_negotiation.cc:769] Sending connection context
0810 00:48:36.769926 (+ 288us) client_negotiation.cc:240] Negotiation successful
0810 00:48:36.770476 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13966,"mutex_wait_us":469,"spinlock_wait_cycles":33024,"thread_start_us":14253,"threads_started":1}
I20260810 00:48:36.771014 19313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:36.732136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:52265 (local address 127.28.6.2:46419)
0810 00:48:36.745075 (+ 12939us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:36.745101 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:36.746148 (+ 1047us) server_negotiation.cc:407] Connection header received
0810 00:48:36.749382 (+ 3234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:36.749391 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:36.749829 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:36.750197 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:36.753221 (+ 3024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:36.755155 (+ 1934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:36.758015 (+ 2860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:36.759117 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:36.759311 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:36.763894 (+ 4583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:36.763908 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:36.763929 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:36.763980 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:36.767133 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:36.767961 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:36.767971 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:36.767981 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:36.768183 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:36.768664 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:36.768671 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:36.768674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:36.769169 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:36.769356 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:36.770165 (+ 809us) server_negotiation.cc:299] Negotiation successful
0810 00:48:36.770526 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12637,"thread_start_us":12928,"threads_started":1}
I20260810 00:48:36.771747 19312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:36.732261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:58733 (local address 127.28.6.3:39599)
0810 00:48:36.747089 (+ 14828us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:36.747102 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:36.747139 (+ 37us) server_negotiation.cc:407] Connection header received
0810 00:48:36.749329 (+ 2190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:36.749343 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:36.749829 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:36.750197 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:36.753229 (+ 3032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:36.755155 (+ 1926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:36.758015 (+ 2860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:36.759161 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:36.759342 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:36.764074 (+ 4732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:36.764081 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:36.764093 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:36.764127 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:36.767336 (+ 3209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:36.768072 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:36.768078 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:36.768084 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:36.768268 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:36.768699 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:36.768702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:36.768704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:36.769161 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:36.769348 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:36.770153 (+ 805us) server_negotiation.cc:299] Negotiation successful
0810 00:48:36.770800 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14668,"thread_start_us":14539,"threads_started":1}
I20260810 00:48:36.771515 19309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:36.731771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:39599 (local address 127.28.6.1:58733)
0810 00:48:36.746285 (+ 14514us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:36.746300 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:36.746732 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:36.750410 (+ 3678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:36.750421 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:36.751745 (+ 1324us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:48:36.752990 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:36.753027 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:36.755374 (+ 2347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:36.755380 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:36.757861 (+ 2481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:36.757896 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:36.759519 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:36.759533 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:36.759962 (+ 429us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:36.761039 (+ 1077us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:48:36.761059 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:48:36.763790 (+ 2731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:48:36.767471 (+ 3681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:36.767475 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:36.767477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:36.767916 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:36.768405 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:36.768408 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:36.768411 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:36.768561 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:36.769317 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:48:36.769325 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:48:36.769640 (+ 315us) client_negotiation.cc:769] Sending connection context
0810 00:48:36.769925 (+ 285us) client_negotiation.cc:240] Negotiation successful
0810 00:48:36.770621 (+ 696us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14402,"mutex_wait_us":630,"spinlock_wait_cycles":72320,"thread_start_us":13738,"threads_started":1}
W20260810 00:48:36.774802 18685 leader_election.cc:341] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e7878590bbf54ded8185416557742988 (127.28.6.3:39599): Not found: Tablet not found: 7c968565f23144889049a1b39b75c090
W20260810 00:48:36.775141 18683 leader_election.cc:341] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d3b6361ac1dd42839442096b8b99e033 (127.28.6.2:46419): Not found: Tablet not found: 7c968565f23144889049a1b39b75c090
I20260810 00:48:36.775329 18683 leader_election.cc:302] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5baea45d390a4f0ab315bcf66cf1687a; no voters: d3b6361ac1dd42839442096b8b99e033, e7878590bbf54ded8185416557742988
I20260810 00:48:36.775759 19308 raft_consensus.cc:2726] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 00:48:37.077322 19316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:37.076312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.3:38081)
0810 00:48:37.076873 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:37.077039 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":300,"threads_started":1}
I20260810 00:48:37.171896 19311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:37.171370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.1:45199)
0810 00:48:37.171592 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:37.171696 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 00:48:37.627743 19317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:37.626984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.2:53415)
0810 00:48:37.627436 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:37.627538 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":232,"threads_started":1}
I20260810 00:48:38.078812 19319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:38.078158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.3:56783)
0810 00:48:38.078573 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:38.078649 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":229,"threads_started":1}
I20260810 00:48:38.173458 19320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:38.172714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.1:33287)
0810 00:48:38.173198 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:38.173274 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":221,"threads_started":1}
I20260810 00:48:38.377032 19322 raft_consensus.cc:482] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:48:38.377311 19322 raft_consensus.cc:504] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7878590bbf54ded8185416557742988" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39599 } } peers { permanent_uuid: "5baea45d390a4f0ab315bcf66cf1687a" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 39615 } } peers { permanent_uuid: "d3b6361ac1dd42839442096b8b99e033" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46419 } }
I20260810 00:48:38.377969 19322 leader_election.cc:288] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e7878590bbf54ded8185416557742988 (127.28.6.3:39599), d3b6361ac1dd42839442096b8b99e033 (127.28.6.2:46419)
W20260810 00:48:38.379112 18683 leader_election.cc:341] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d3b6361ac1dd42839442096b8b99e033 (127.28.6.2:46419): Not found: Tablet not found: 7c968565f23144889049a1b39b75c090
W20260810 00:48:38.379312 18685 leader_election.cc:341] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e7878590bbf54ded8185416557742988 (127.28.6.3:39599): Not found: Tablet not found: 7c968565f23144889049a1b39b75c090
I20260810 00:48:38.379482 18685 leader_election.cc:302] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5baea45d390a4f0ab315bcf66cf1687a; no voters: d3b6361ac1dd42839442096b8b99e033, e7878590bbf54ded8185416557742988
I20260810 00:48:38.379801 19322 raft_consensus.cc:2726] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 00:48:38.629294 19323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:38.628604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.2:41217)
0810 00:48:38.629032 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:38.629107 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":227,"threads_started":1}
I20260810 00:48:39.080358 19324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:39.079691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.3:45243)
0810 00:48:39.080111 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:39.080187 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":221,"threads_started":1}
I20260810 00:48:39.175092 19326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:39.174390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.1:53721)
0810 00:48:39.174829 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:39.174931 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":238,"threads_started":1}
I20260810 00:48:39.630847 19327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:39.630157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.2:37349)
0810 00:48:39.630590 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:39.630673 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":202,"threads_started":1}
I20260810 00:48:40.081871 19328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:40.081176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.3:33093)
0810 00:48:40.081606 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:40.081685 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":216,"threads_started":1}
I20260810 00:48:40.176927 19330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:40.176189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.1:44509)
0810 00:48:40.176633 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:40.176733 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":243,"threads_started":1}
I20260810 00:48:40.632486 19331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:40.631811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.2:46549)
0810 00:48:40.632251 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:40.632327 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":228,"threads_started":1}
I20260810 00:48:41.083564 19332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:41.082819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.3:52989)
0810 00:48:41.083331 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:41.083409 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":293,"threads_started":1}
I20260810 00:48:41.178500 19333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:41.177837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.1:51627)
0810 00:48:41.178258 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:41.178336 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":221,"threads_started":1}
I20260810 00:48:41.462754 19334 raft_consensus.cc:482] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:48:41.463055 19334 raft_consensus.cc:504] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7878590bbf54ded8185416557742988" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39599 } } peers { permanent_uuid: "5baea45d390a4f0ab315bcf66cf1687a" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 39615 } } peers { permanent_uuid: "d3b6361ac1dd42839442096b8b99e033" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46419 } }
I20260810 00:48:41.463794 19334 leader_election.cc:288] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e7878590bbf54ded8185416557742988 (127.28.6.3:39599), d3b6361ac1dd42839442096b8b99e033 (127.28.6.2:46419)
W20260810 00:48:41.464741 18685 leader_election.cc:341] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e7878590bbf54ded8185416557742988 (127.28.6.3:39599): Not found: Tablet not found: 7c968565f23144889049a1b39b75c090
W20260810 00:48:41.464948 18683 leader_election.cc:341] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d3b6361ac1dd42839442096b8b99e033 (127.28.6.2:46419): Not found: Tablet not found: 7c968565f23144889049a1b39b75c090
I20260810 00:48:41.465107 18683 leader_election.cc:302] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5baea45d390a4f0ab315bcf66cf1687a; no voters: d3b6361ac1dd42839442096b8b99e033, e7878590bbf54ded8185416557742988
I20260810 00:48:41.465368 19334 raft_consensus.cc:2726] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 00:48:41.633962 19335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:41.633249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.2:46423)
0810 00:48:41.633698 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:41.633781 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":241,"threads_started":1}
I20260810 00:48:42.085137 19336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:42.084424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.3:54319)
0810 00:48:42.084899 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:42.084972 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":259,"threads_started":1}
I20260810 00:48:42.180034 19339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:42.179319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.1:43553)
0810 00:48:42.179753 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:42.179836 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":225,"threads_started":1}
I20260810 00:48:42.635694 19340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:42.634832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.2:49767)
0810 00:48:42.635441 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:42.635514 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":315,"threads_started":1}
I20260810 00:48:43.086771 19341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:43.086024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.3:43409)
0810 00:48:43.086498 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:43.086572 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":234,"threads_started":1}
I20260810 00:48:43.181557 19342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:43.180862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.1:42995)
0810 00:48:43.181310 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:43.181392 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":235,"threads_started":1}
I20260810 00:48:43.637212 19343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:43.636549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.2:59655)
0810 00:48:43.636958 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:43.637030 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":214,"threads_started":1}
I20260810 00:48:44.088297 19344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:44.087616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.3:45371)
0810 00:48:44.088041 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:44.088114 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":232,"threads_started":1}
I20260810 00:48:44.183215 19345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:44.182437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.1:44377)
0810 00:48:44.182868 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:44.183040 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":221,"threads_started":1}
I20260810 00:48:44.638803 19348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:44.638085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.2:49373)
0810 00:48:44.638551 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:44.638633 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":226,"threads_started":1}
I20260810 00:48:44.847417 19353 raft_consensus.cc:482] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:48:44.847760 19353 raft_consensus.cc:504] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7878590bbf54ded8185416557742988" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39599 } } peers { permanent_uuid: "5baea45d390a4f0ab315bcf66cf1687a" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 39615 } } peers { permanent_uuid: "d3b6361ac1dd42839442096b8b99e033" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 46419 } }
I20260810 00:48:44.848614 19353 leader_election.cc:288] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e7878590bbf54ded8185416557742988 (127.28.6.3:39599), d3b6361ac1dd42839442096b8b99e033 (127.28.6.2:46419)
W20260810 00:48:44.849622 18683 leader_election.cc:341] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d3b6361ac1dd42839442096b8b99e033 (127.28.6.2:46419): Not found: Tablet not found: 7c968565f23144889049a1b39b75c090
W20260810 00:48:44.849824 18685 leader_election.cc:341] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e7878590bbf54ded8185416557742988 (127.28.6.3:39599): Not found: Tablet not found: 7c968565f23144889049a1b39b75c090
I20260810 00:48:44.849931 18685 leader_election.cc:302] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5baea45d390a4f0ab315bcf66cf1687a; no voters: d3b6361ac1dd42839442096b8b99e033, e7878590bbf54ded8185416557742988
I20260810 00:48:44.850160 19353 raft_consensus.cc:2726] T 7c968565f23144889049a1b39b75c090 P 5baea45d390a4f0ab315bcf66cf1687a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 00:48:45.089919 19355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:45.089226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.3:46417)
0810 00:48:45.089679 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:45.089762 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":232,"threads_started":1}
I20260810 00:48:45.184978 19357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:45.184167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.1:32799)
0810 00:48:45.184663 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:45.184758 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":261,"threads_started":1}
I20260810 00:48:45.640300 19358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:45.639642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.2:54847)
0810 00:48:45.640062 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:45.640134 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":227,"threads_started":1}
I20260810 00:48:46.091522 19359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:46.090799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.3:42607)
0810 00:48:46.091275 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:46.091351 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":285,"threads_started":1}
I20260810 00:48:46.186565 19361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:46.185806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.1:60215)
0810 00:48:46.186321 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:46.186397 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":284,"threads_started":1}
I20260810 00:48:46.641903 19362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:46.641152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.2:57177)
0810 00:48:46.641649 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:46.641736 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":268,"threads_started":1}
I20260810 00:48:47.093093 19363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:47.092395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.3:46377)
0810 00:48:47.092866 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:47.092944 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":247,"threads_started":1}
I20260810 00:48:47.188226 19366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:47.187544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.1:38917)
0810 00:48:47.187963 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:47.188043 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":227,"threads_started":1}
I20260810 00:48:47.643637 19367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:47.642951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.2:58833)
0810 00:48:47.643390 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:47.643463 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":217,"threads_started":1}
I20260810 00:48:48.094695 19368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:48.094021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.3:34433)
0810 00:48:48.094453 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:48.094527 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":229,"threads_started":1}
I20260810 00:48:48.189884 19371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:48.189130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.1:50051)
0810 00:48:48.189605 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:48.189700 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":262,"threads_started":1}
I20260810 00:48:48.645190 19372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:48.644492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.2:38149)
0810 00:48:48.644946 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:48.645017 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":237,"threads_started":1}
I20260810 00:48:49.096330 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:49.095574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.3:41153)
0810 00:48:49.096086 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:49.096162 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":257,"threads_started":1}
I20260810 00:48:49.191517 19375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:49.190756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.1:49615)
0810 00:48:49.191252 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:49.191340 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":287,"threads_started":1}
I20260810 00:48:49.646689 19378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:49.645994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.2:38791)
0810 00:48:49.646424 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:49.646496 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":229,"threads_started":1}
I20260810 00:48:50.097909 19380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:50.097179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.3:47013)
0810 00:48:50.097639 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:50.097721 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":236,"threads_started":1}
I20260810 00:48:50.193423 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:50.192643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.1:58111)
0810 00:48:50.193144 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:50.193220 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":259,"threads_started":1}
I20260810 00:48:50.648442 19383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:50.647760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.2:44191)
0810 00:48:50.648207 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:50.648288 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":234,"threads_started":1}
I20260810 00:48:51.078986 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 18592
I20260810 00:48:51.093218 19384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:51.080100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47890 (local address 127.28.6.1:39615)
0810 00:48:51.080775 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:51.080782 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:51.080809 (+ 27us) server_negotiation.cc:407] Connection header received
0810 00:48:51.080868 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:51.080875 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:51.081100 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:51.081244 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:51.082565 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:51.083656 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:51.084875 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:51.085457 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:51.085551 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:51.088073 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:51.088083 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:51.088097 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:51.088131 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:51.090775 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:51.091443 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:51.091450 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:51.091452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:51.091589 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:51.091928 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:51.091931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:51.091933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:51.092282 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:51.092515 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:51.092756 (+ 241us) server_negotiation.cc:299] Negotiation successful
0810 00:48:51.092945 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":387,"threads_started":1}
I20260810 00:48:51.094408 18854 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:48:51.099686 19387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:51.098964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:36385 (local address 127.28.6.3:46749)
0810 00:48:51.099452 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:51.099542 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":270,"threads_started":1}
I20260810 00:48:51.183844 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 18592
I20260810 00:48:51.210309 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 18864
I20260810 00:48:51.224400 19388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:51.211113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59662 (local address 127.28.6.2:46419)
0810 00:48:51.211638 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:51.211642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:51.211659 (+ 17us) server_negotiation.cc:407] Connection header received
0810 00:48:51.211715 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:51.211718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:51.211786 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:51.211878 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:51.213284 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:51.214102 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:51.215505 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:51.215929 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:51.216025 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:51.219186 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:51.219191 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:51.219195 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:51.219226 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:51.221922 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:51.222568 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:51.222572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:51.222574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:51.222690 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:51.223106 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:51.223113 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:51.223118 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:51.223405 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:51.223643 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:51.223838 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 00:48:51.223995 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":300,"threads_started":1}
I20260810 00:48:51.225132 19062 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:48:51.307528 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 18864
I20260810 00:48:51.333132 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 19080
I20260810 00:48:51.346150 19392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:51.333918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53222 (local address 127.28.6.3:39599)
0810 00:48:51.334532 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:51.334537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:51.334554 (+ 17us) server_negotiation.cc:407] Connection header received
0810 00:48:51.334612 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:51.334615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:51.334685 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:51.334783 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:51.336213 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:51.337019 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:51.338299 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:51.338695 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:51.338784 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:51.341381 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:51.341386 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:51.341389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:51.341418 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:51.344028 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:51.344637 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:51.344641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:51.344643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:51.344795 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:51.345099 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:51.345102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:51.345104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:51.345337 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:51.345541 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:51.345707 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 00:48:51.345862 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":349,"threads_started":1}
I20260810 00:48:51.346814 19286 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:48:51.428546 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 19080
2026-08-10T00:48:51Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24824 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:48:51 dist-test-slave-1nl6 krb5kdc[19398](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:48:51 dist-test-slave-1nl6 krb5kdc[19398](info): set up 2 sockets
Aug 10 00:48:51 dist-test-slave-1nl6 krb5kdc[19398](info): commencing operation
krb5kdc: starting...
W20260810 00:48:52.731628 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.255s user 0.001s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:48:52 dist-test-slave-1nl6 krb5kdc[19398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:48:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:48:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.6.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
I20260810 00:48:52.985850 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:44053
--webserver_interface=127.28.6.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:39585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:44053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:48:53.322611 19419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:48:53.322949 19419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:48:53.323134 19419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:48:53.330205 19419 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 00:48:53.330331 19419 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 00:48:53.330425 19419 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 00:48:53.330520 19419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:48:53.330619 19419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:48:53.330718 19419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:48:53.330788 19419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:48:53.330865 19419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:48:53.340358 19419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:39585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:44053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:44053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:48:53.341616 19419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:48:53.342958 19419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:48:53.355154 19435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:53.355278 19431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:53.355229 19432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:48:53.356079 19419 server_base.cc:1034] running on GCE node
I20260810 00:48:53.356889 19419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:48:53.358278 19419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:48:53.359522 19419 hybrid_clock.cc:648] HybridClock initialized: now 1786322933359465 us; error 33 us; skew 500 ppm
I20260810 00:48:53.359848 19419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:48:53 dist-test-slave-1nl6 krb5kdc[19398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322933, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:48:53.363189 19419 init.cc:377] Logged in from keytab as kudu/127.28.6.62@KRBTEST.COM (short username kudu)
I20260810 00:48:53.364650 19419 webserver.cc:460] Webserver started at http://127.28.6.62:36879/ using document root <none> and password file <none>
I20260810 00:48:53.365353 19419 fs_manager.cc:359] Metadata directory not provided
I20260810 00:48:53.365497 19419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:48:53.365829 19419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:48:53.368925 19419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "491fe8c474e84bd9b275042408d6adbb"
format_stamp: "Formatted at 2026-08-10 00:48:53 on dist-test-slave-1nl6"
server_key: "9782b27bfcedfd6c8521961244e154bf"
server_key_iv: "677b1b6c806eb2c5e6812568cbcc18f1"
server_key_version: "encryptionkey@0"
I20260810 00:48:53.369714 19419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "491fe8c474e84bd9b275042408d6adbb"
format_stamp: "Formatted at 2026-08-10 00:48:53 on dist-test-slave-1nl6"
server_key: "9782b27bfcedfd6c8521961244e154bf"
server_key_iv: "677b1b6c806eb2c5e6812568cbcc18f1"
server_key_version: "encryptionkey@0"
I20260810 00:48:53.374739 19419 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 00:48:53.378261 19446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:53.379531 19419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 00:48:53.379734 19419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "491fe8c474e84bd9b275042408d6adbb"
format_stamp: "Formatted at 2026-08-10 00:48:53 on dist-test-slave-1nl6"
server_key: "9782b27bfcedfd6c8521961244e154bf"
server_key_iv: "677b1b6c806eb2c5e6812568cbcc18f1"
server_key_version: "encryptionkey@0"
I20260810 00:48:53.379962 19419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:48:53.419391 19419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:48:53.423415 19419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:48:53.423695 19419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:48:53.442317 19419 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:44053
I20260810 00:48:53.442351 19528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:44053 every 8 connection(s)
I20260810 00:48:53.443744 19419 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:48:53.447378 19529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:48:53.449640 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 19419
I20260810 00:48:53.449882 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:48:53.450536 28696 external_mini_cluster.cc:1428] Setting key bda89851d6c7d746af0bbc386ecb7e95
I20260810 00:48:53.460333 19529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb: Bootstrap starting.
I20260810 00:48:53.463832 19529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb: Neither blocks nor log segments found. Creating new log.
I20260810 00:48:53.464777 19529 log.cc:824] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb: Log is configured to *not* fsync() on all Append() calls
Aug 10 00:48:53 dist-test-slave-1nl6 krb5kdc[19398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:48:53.467803 19529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb: No bootstrap required, opened a new log
I20260810 00:48:53.474377 19534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:53.452652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60736 (local address 127.28.6.62:44053)
0810 00:48:53.453469 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:53.453486 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:53.453537 (+ 51us) server_negotiation.cc:407] Connection header received
0810 00:48:53.455914 (+ 2377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:53.455945 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:53.456427 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:53.456851 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:53.458208 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:53.460142 (+ 1934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:53.461544 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:53.462624 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:53.462805 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:53.467125 (+ 4320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:53.467133 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:53.467147 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:53.467196 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:53.470217 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:53.471056 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:53.471062 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:53.471069 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:53.471270 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:53.471700 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:53.471704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:53.471706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:53.472227 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:53.472455 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:53.472953 (+ 498us) server_negotiation.cc:299] Negotiation successful
0810 00:48:53.473351 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":318,"threads_started":1}
I20260810 00:48:53.477068 19529 raft_consensus.cc:348] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491fe8c474e84bd9b275042408d6adbb" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 44053 } }
I20260810 00:48:53.477437 19529 raft_consensus.cc:374] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:48:53.477577 19529 raft_consensus.cc:729] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491fe8c474e84bd9b275042408d6adbb, State: Initialized, Role: FOLLOWER
I20260810 00:48:53.478091 19529 consensus_queue.cc:260] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491fe8c474e84bd9b275042408d6adbb" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 44053 } }
I20260810 00:48:53.478344 19529 raft_consensus.cc:388] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:48:53.478487 19529 raft_consensus.cc:482] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:48:53.478663 19529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:48:53.480825 19529 raft_consensus.cc:504] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491fe8c474e84bd9b275042408d6adbb" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 44053 } }
I20260810 00:48:53.481302 19529 leader_election.cc:302] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 491fe8c474e84bd9b275042408d6adbb; no voters:
I20260810 00:48:53.481884 19529 leader_election.cc:288] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:48:53.482051 19536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:48:53.482798 19536 raft_consensus.cc:686] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [term 1 LEADER]: Becoming Leader. State: Replica: 491fe8c474e84bd9b275042408d6adbb, State: Running, Role: LEADER
I20260810 00:48:53.483501 19536 consensus_queue.cc:237] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491fe8c474e84bd9b275042408d6adbb" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 44053 } }
I20260810 00:48:53.484354 19529 sys_catalog.cc:564] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:48:53.486636 19537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "491fe8c474e84bd9b275042408d6adbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491fe8c474e84bd9b275042408d6adbb" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 44053 } } }
I20260810 00:48:53.486662 19538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 491fe8c474e84bd9b275042408d6adbb. Latest consensus state: current_term: 1 leader_uuid: "491fe8c474e84bd9b275042408d6adbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491fe8c474e84bd9b275042408d6adbb" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 44053 } } }
I20260810 00:48:53.487110 19538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [sys.catalog]: This master's current role is: LEADER
I20260810 00:48:53.487108 19537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [sys.catalog]: This master's current role is: LEADER
I20260810 00:48:53.490783 19547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:48:53.496376 19547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:48:53.505999 19547 catalog_manager.cc:1347] Generated new cluster ID: 8e4bca419786490cbc821bfe13a74ebc
I20260810 00:48:53.506129 19547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:48:53.517086 19547 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:48:53.518438 19547 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:48:53.525210 19547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb: Generated new TSK 0
I20260810 00:48:53.525998 19547 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.6.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab.
Entry for principal kudu/127.28.6.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab.
I20260810 00:48:53.596380 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.1:0
--local_ip_for_outbound_sockets=127.28.6.1
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:44053
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:39585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:48:53.932548 19570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:48:53.932868 19570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:48:53.933041 19570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:48:53.940654 19570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:48:53.940811 19570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:48:53.940982 19570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.1
I20260810 00:48:53.951088 19570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:39585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:44053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.1
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:48:53.952492 19570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:48:53.953788 19570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:48:53.966661 19580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:53.966687 19587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:53.966792 19581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:48:53.967996 19570 server_base.cc:1034] running on GCE node
I20260810 00:48:53.968576 19570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:48:53.969486 19570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:48:53.970690 19570 hybrid_clock.cc:648] HybridClock initialized: now 1786322933970646 us; error 43 us; skew 500 ppm
I20260810 00:48:53.971048 19570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:48:53 dist-test-slave-1nl6 krb5kdc[19398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322933, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:48:53.974272 19570 init.cc:377] Logged in from keytab as kudu/127.28.6.1@KRBTEST.COM (short username kudu)
I20260810 00:48:53.975809 19570 webserver.cc:460] Webserver started at http://127.28.6.1:46097/ using document root <none> and password file <none>
I20260810 00:48:53.976446 19570 fs_manager.cc:359] Metadata directory not provided
I20260810 00:48:53.976548 19570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:48:53.976863 19570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:48:53.979859 19570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/data/instance:
uuid: "92f2b001cfd2415dba06994af259aabb"
format_stamp: "Formatted at 2026-08-10 00:48:53 on dist-test-slave-1nl6"
server_key: "d41800cbef261e62d0c2da089016b595"
server_key_iv: "801c02d3a8d6bf5d64dcc54e4375f125"
server_key_version: "encryptionkey@0"
I20260810 00:48:53.980572 19570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance:
uuid: "92f2b001cfd2415dba06994af259aabb"
format_stamp: "Formatted at 2026-08-10 00:48:53 on dist-test-slave-1nl6"
server_key: "d41800cbef261e62d0c2da089016b595"
server_key_iv: "801c02d3a8d6bf5d64dcc54e4375f125"
server_key_version: "encryptionkey@0"
I20260810 00:48:53.986001 19570 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 00:48:53.989425 19595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:53.990697 19570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 00:48:53.990873 19570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/wal
uuid: "92f2b001cfd2415dba06994af259aabb"
format_stamp: "Formatted at 2026-08-10 00:48:53 on dist-test-slave-1nl6"
server_key: "d41800cbef261e62d0c2da089016b595"
server_key_iv: "801c02d3a8d6bf5d64dcc54e4375f125"
server_key_version: "encryptionkey@0"
I20260810 00:48:53.991110 19570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:48:54.037555 19570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:48:54.041671 19570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:48:54.041973 19570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:48:54.043004 19570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:48:54.044322 19570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:48:54.044423 19570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:54.044572 19570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:48:54.044646 19570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:54.077538 19570 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.1:35567
I20260810 00:48:54.077581 19786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.1:35567 every 8 connection(s)
I20260810 00:48:54.078796 19570 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
I20260810 00:48:54.084277 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 19570
I20260810 00:48:54.084491 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance
I20260810 00:48:54.085409 28696 external_mini_cluster.cc:1428] Setting key fe322ae1c50c3448fae8f022ba3c9fbf
Aug 10 00:48:54 dist-test-slave-1nl6 krb5kdc[19398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322933, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:48:54.101212 19790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:54.081307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:49149 (local address 127.28.6.62:44053)
0810 00:48:54.081763 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:54.081767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:54.082688 (+ 921us) server_negotiation.cc:407] Connection header received
0810 00:48:54.085208 (+ 2520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:54.085214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:54.085299 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:54.085396 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:54.087293 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:54.088140 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:54.090650 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:54.091173 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:54.091278 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:54.095656 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:54.095661 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:54.095665 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:54.095696 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:54.098167 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:54.098864 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:54.098869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:54.098871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:54.099045 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:54.099485 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:54.099490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:54.099493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:54.099768 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:54.099924 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:54.100621 (+ 697us) server_negotiation.cc:299] Negotiation successful
0810 00:48:54.100847 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":225,"threads_started":1}
I20260810 00:48:54.102147 19794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:54.081825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.1:49149)
0810 00:48:54.082510 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:54.082575 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:54.084875 (+ 2300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:54.085684 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:54.085694 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:54.086397 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:48:54.087120 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:54.087140 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:54.088323 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:54.088329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:54.090511 (+ 2182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:54.090529 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:54.091418 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:54.091428 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:54.091765 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:54.092515 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:48:54.092548 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:48:54.095404 (+ 2856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:48:54.098309 (+ 2905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:54.098317 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:54.098332 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:54.098721 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:54.099157 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:54.099162 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:54.099165 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:54.099364 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:54.099900 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:48:54.099907 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:48:54.100228 (+ 321us) client_negotiation.cc:769] Sending connection context
0810 00:48:54.100494 (+ 266us) client_negotiation.cc:240] Negotiation successful
0810 00:48:54.101021 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":269,"threads_started":1}
I20260810 00:48:54.103720 19787 heartbeater.cc:347] Connected to a master server at 127.28.6.62:44053
I20260810 00:48:54.104017 19787 heartbeater.cc:464] Registering TS with master...
I20260810 00:48:54.104705 19787 heartbeater.cc:511] Master 127.28.6.62:44053 requested a full tablet report, sending...
I20260810 00:48:54.106837 19475 ts_manager.cc:194] Registered new tserver with Master: 92f2b001cfd2415dba06994af259aabb (127.28.6.1:35567)
I20260810 00:48:54.108397 19475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.1@KRBTEST.COM'} at 127.28.6.1:49149
WARNING: no policy specified for kudu/127.28.6.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab.
Entry for principal kudu/127.28.6.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab.
I20260810 00:48:54.149328 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.2:0
--local_ip_for_outbound_sockets=127.28.6.2
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:44053
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:39585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:48:54.489527 19799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:48:54.489794 19799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:48:54.489890 19799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:48:54.497219 19799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:48:54.497326 19799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:48:54.497444 19799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.2
I20260810 00:48:54.506851 19799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:39585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:44053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.2
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:48:54.508074 19799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:48:54.509387 19799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:48:54.522724 19807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:54.522630 19808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:54.522668 19811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:48:54.523439 19799 server_base.cc:1034] running on GCE node
I20260810 00:48:54.524027 19799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:48:54.524917 19799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:48:54.526122 19799 hybrid_clock.cc:648] HybridClock initialized: now 1786322934526070 us; error 42 us; skew 500 ppm
I20260810 00:48:54.526466 19799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:48:54 dist-test-slave-1nl6 krb5kdc[19398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322934, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:48:54.534364 19799 init.cc:377] Logged in from keytab as kudu/127.28.6.2@KRBTEST.COM (short username kudu)
I20260810 00:48:54.535893 19799 webserver.cc:460] Webserver started at http://127.28.6.2:41307/ using document root <none> and password file <none>
I20260810 00:48:54.536592 19799 fs_manager.cc:359] Metadata directory not provided
I20260810 00:48:54.536743 19799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:48:54.537060 19799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:48:54.540050 19799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/data/instance:
uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea"
format_stamp: "Formatted at 2026-08-10 00:48:54 on dist-test-slave-1nl6"
server_key: "4930f94ecea10c721337ac33af729447"
server_key_iv: "e875c1dc1d9b0dea9f7e634fd9211dd5"
server_key_version: "encryptionkey@0"
I20260810 00:48:54.540908 19799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance:
uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea"
format_stamp: "Formatted at 2026-08-10 00:48:54 on dist-test-slave-1nl6"
server_key: "4930f94ecea10c721337ac33af729447"
server_key_iv: "e875c1dc1d9b0dea9f7e634fd9211dd5"
server_key_version: "encryptionkey@0"
I20260810 00:48:54.546065 19799 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260810 00:48:54.549427 19820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:54.550608 19799 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 00:48:54.550793 19799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/wal
uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea"
format_stamp: "Formatted at 2026-08-10 00:48:54 on dist-test-slave-1nl6"
server_key: "4930f94ecea10c721337ac33af729447"
server_key_iv: "e875c1dc1d9b0dea9f7e634fd9211dd5"
server_key_version: "encryptionkey@0"
I20260810 00:48:54.551029 19799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:48:54.570379 19799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:48:54.574437 19799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:48:54.574730 19799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:48:54.575783 19799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:48:54.577309 19799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:48:54.577425 19799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:54.577584 19799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:48:54.577656 19799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:54.613934 19799 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.2:34761
I20260810 00:48:54.614019 20009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.2:34761 every 8 connection(s)
I20260810 00:48:54.615289 19799 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
I20260810 00:48:54.615607 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 19799
I20260810 00:48:54.615808 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance
I20260810 00:48:54.616447 28696 external_mini_cluster.cc:1428] Setting key 631ad364e48b2658391d86198558be6d
Aug 10 00:48:54 dist-test-slave-1nl6 krb5kdc[19398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322934, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:48:54.638975 20012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:54.618094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:51707 (local address 127.28.6.62:44053)
0810 00:48:54.618569 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:54.618576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:54.619791 (+ 1215us) server_negotiation.cc:407] Connection header received
0810 00:48:54.622698 (+ 2907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:54.622702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:54.622766 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:54.622867 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:54.625115 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:54.625805 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:54.628431 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:54.628849 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:54.628950 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:54.633181 (+ 4231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:54.633187 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:54.633190 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:54.633223 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:54.635874 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:54.636593 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:54.636598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:54.636604 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:54.636740 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:54.637196 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:54.637200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:54.637202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:54.637495 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:54.637662 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:54.638357 (+ 695us) server_negotiation.cc:299] Negotiation successful
0810 00:48:54.638574 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":238,"threads_started":1}
I20260810 00:48:54.639883 20016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:54.618709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.2:51707)
0810 00:48:54.619597 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:54.619646 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:54.622413 (+ 2767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:54.623196 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:54.623209 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:54.624068 (+ 859us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:48:54.624927 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:54.624957 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:54.625971 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:54.625979 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:54.628165 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:54.628196 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:54.629094 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:54.629104 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:54.629433 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:54.630185 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:48:54.630227 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:48:54.632943 (+ 2716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:48:54.636018 (+ 3075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:54.636026 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:54.636042 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:54.636463 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:54.636858 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:54.636863 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:54.636866 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:54.637067 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:54.637640 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:48:54.637648 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:48:54.637977 (+ 329us) client_negotiation.cc:769] Sending connection context
0810 00:48:54.638244 (+ 267us) client_negotiation.cc:240] Negotiation successful
0810 00:48:54.638741 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":363,"threads_started":1}
I20260810 00:48:54.641510 20010 heartbeater.cc:347] Connected to a master server at 127.28.6.62:44053
I20260810 00:48:54.641820 20010 heartbeater.cc:464] Registering TS with master...
I20260810 00:48:54.642544 20010 heartbeater.cc:511] Master 127.28.6.62:44053 requested a full tablet report, sending...
I20260810 00:48:54.644125 19475 ts_manager.cc:194] Registered new tserver with Master: 92c7bf0b0c8a4ace9fed6b4576ddc4ea (127.28.6.2:34761)
I20260810 00:48:54.644907 19475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.2@KRBTEST.COM'} at 127.28.6.2:51707
WARNING: no policy specified for kudu/127.28.6.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab.
Entry for principal kudu/127.28.6.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab.
I20260810 00:48:54.680416 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.3:0
--local_ip_for_outbound_sockets=127.28.6.3
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:44053
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:39585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:48:55.019193 20019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:48:55.019470 20019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:48:55.019583 20019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:48:55.026705 20019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:48:55.026813 20019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:48:55.026948 20019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.3
I20260810 00:48:55.036536 20019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:39585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:44053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.3
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:48:55.037814 20019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:48:55.039273 20019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:48:55.052665 20030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:55.052712 20028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:48:55.052743 20027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:48:55.053857 20019 server_base.cc:1034] running on GCE node
I20260810 00:48:55.054670 20019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:48:55.055774 20019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:48:55.057024 20019 hybrid_clock.cc:648] HybridClock initialized: now 1786322935056950 us; error 57 us; skew 500 ppm
I20260810 00:48:55.057359 20019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:48:55 dist-test-slave-1nl6 krb5kdc[19398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322935, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:48:55.060551 20019 init.cc:377] Logged in from keytab as kudu/127.28.6.3@KRBTEST.COM (short username kudu)
I20260810 00:48:55.062147 20019 webserver.cc:460] Webserver started at http://127.28.6.3:39153/ using document root <none> and password file <none>
I20260810 00:48:55.062839 20019 fs_manager.cc:359] Metadata directory not provided
I20260810 00:48:55.063019 20019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:48:55.063341 20019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:48:55.066221 20019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/data/instance:
uuid: "c2aa322985374b8092a1f4adb947ab42"
format_stamp: "Formatted at 2026-08-10 00:48:55 on dist-test-slave-1nl6"
server_key: "182ae569985ac8b68141a5764dd00b01"
server_key_iv: "b7fce0030639b16e31f577301e6c7789"
server_key_version: "encryptionkey@0"
I20260810 00:48:55.067108 20019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance:
uuid: "c2aa322985374b8092a1f4adb947ab42"
format_stamp: "Formatted at 2026-08-10 00:48:55 on dist-test-slave-1nl6"
server_key: "182ae569985ac8b68141a5764dd00b01"
server_key_iv: "b7fce0030639b16e31f577301e6c7789"
server_key_version: "encryptionkey@0"
I20260810 00:48:55.072252 20019 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 00:48:55.075610 20038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:55.076854 20019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 00:48:55.077064 20019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/wal
uuid: "c2aa322985374b8092a1f4adb947ab42"
format_stamp: "Formatted at 2026-08-10 00:48:55 on dist-test-slave-1nl6"
server_key: "182ae569985ac8b68141a5764dd00b01"
server_key_iv: "b7fce0030639b16e31f577301e6c7789"
server_key_version: "encryptionkey@0"
I20260810 00:48:55.077296 20019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:48:55.095955 20019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:48:55.099805 20019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:48:55.100112 20019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:48:55.101037 20019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:48:55.102345 20019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:48:55.102450 20019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:55.102608 20019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:48:55.102689 20019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:48:55.111229 19787 heartbeater.cc:503] Master 127.28.6.62:44053 was elected leader, sending a full tablet report...
I20260810 00:48:55.136972 20019 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.3:46009
I20260810 00:48:55.137089 20226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.3:46009 every 8 connection(s)
I20260810 00:48:55.138289 20019 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
I20260810 00:48:55.146245 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 20019
I20260810 00:48:55.146437 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance
I20260810 00:48:55.146878 28696 external_mini_cluster.cc:1428] Setting key 3200cf43b270e29cab6b8f5c67fa212b
Aug 10 00:48:55 dist-test-slave-1nl6 krb5kdc[19398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322935, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:48:55.158563 20233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:55.140996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:58565 (local address 127.28.6.62:44053)
0810 00:48:55.141502 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:55.141507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:55.142320 (+ 813us) server_negotiation.cc:407] Connection header received
0810 00:48:55.144225 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:55.144231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:55.144335 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:55.144460 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:55.145932 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:55.146836 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:55.148868 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:55.149536 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:55.149676 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:55.153295 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:55.153300 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:55.153304 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:55.153336 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:55.155987 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:55.156719 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:55.156729 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:55.156732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:55.156843 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:55.157214 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:55.157218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:55.157220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:55.157458 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:55.157564 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:55.158153 (+ 589us) server_negotiation.cc:299] Negotiation successful
0810 00:48:55.158309 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":274,"threads_started":1}
I20260810 00:48:55.159528 20234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:55.141424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.3:58565)
0810 00:48:55.142143 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:55.142198 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:55.143946 (+ 1748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:55.144709 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:55.144717 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:55.145206 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:48:55.145754 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:55.145771 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:55.147021 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:55.147025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:55.148575 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:55.148588 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:55.149749 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:55.149774 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:55.150057 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:55.150742 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:48:55.150769 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:48:55.153100 (+ 2331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:48:55.156181 (+ 3081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:55.156187 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:55.156198 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:55.156616 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:55.156935 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:48:55.156938 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:48:55.156940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:48:55.157107 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:48:55.157563 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:48:55.157568 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:48:55.157853 (+ 285us) client_negotiation.cc:769] Sending connection context
0810 00:48:55.158069 (+ 216us) client_negotiation.cc:240] Negotiation successful
0810 00:48:55.158437 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":278,"threads_started":1}
I20260810 00:48:55.161108 20227 heartbeater.cc:347] Connected to a master server at 127.28.6.62:44053
I20260810 00:48:55.161469 20227 heartbeater.cc:464] Registering TS with master...
I20260810 00:48:55.162325 20227 heartbeater.cc:511] Master 127.28.6.62:44053 requested a full tablet report, sending...
I20260810 00:48:55.163807 19475 ts_manager.cc:194] Registered new tserver with Master: c2aa322985374b8092a1f4adb947ab42 (127.28.6.3:46009)
I20260810 00:48:55.164531 19475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.3@KRBTEST.COM'} at 127.28.6.3:58565
I20260810 00:48:55.174531 28696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 00:48:55.191835 20233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:55.179011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60744 (local address 127.28.6.62:44053)
0810 00:48:55.179215 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:55.179223 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:55.179719 (+ 496us) server_negotiation.cc:407] Connection header received
0810 00:48:55.179981 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:55.179985 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:55.180048 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:55.180128 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:55.181652 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:55.182369 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:55.183883 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:55.184441 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:55.184533 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:55.186984 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:55.186989 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:55.186992 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:55.187038 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:55.189380 (+ 2342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:55.190057 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:55.190061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:55.190063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:55.190175 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:55.190484 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:55.190487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:55.190489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:55.190778 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:55.191019 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:55.191394 (+ 375us) server_negotiation.cc:299] Negotiation successful
0810 00:48:55.191556 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 00:48:55.195820 19475 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 00:48:55.202375 19475 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 00:48:55.223893 20251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:55.213996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:46009 (local address 127.0.0.1:60086)
0810 00:48:55.214631 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:55.214694 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:55.214882 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:55.216257 (+ 1375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:55.216267 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:55.216313 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:48:55.216808 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:55.216829 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:55.218656 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:55.218663 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:55.220810 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:55.220826 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:55.222619 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:55.222625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:55.223034 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:55.223091 (+ 57us) client_negotiation.cc:769] Sending connection context
0810 00:48:55.223253 (+ 162us) client_negotiation.cc:240] Negotiation successful
0810 00:48:55.223447 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":4,"thread_start_us":287,"threads_started":1}
I20260810 00:48:55.224763 20255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:55.215140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:34761 (local address 127.0.0.1:58838)
0810 00:48:55.215705 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:55.215723 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:55.215829 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:55.218472 (+ 2643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:55.218476 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:55.218503 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:48:55.218957 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:55.218970 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:55.220827 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:55.220832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:55.222279 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:55.222291 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:55.224184 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:55.224188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:55.224299 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:55.224321 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 00:48:55.224371 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 00:48:55.224449 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":304,"threads_started":1}
I20260810 00:48:55.224833 20252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:55.214339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60086 (local address 127.28.6.3:46009)
0810 00:48:55.215210 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:55.215220 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:55.215257 (+ 37us) server_negotiation.cc:407] Connection header received
0810 00:48:55.215359 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:55.215366 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:55.215624 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:55.215862 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:48:55.217083 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:55.218487 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:55.220969 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:55.222405 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:55.222551 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:55.222810 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:55.224293 (+ 1483us) server_negotiation.cc:299] Negotiation successful
0810 00:48:55.224477 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":463,"threads_started":1}
I20260810 00:48:55.225461 20257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:55.215788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:35567 (local address 127.0.0.1:42850)
0810 00:48:55.216325 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:55.216341 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:55.216464 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:55.217640 (+ 1176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:55.217647 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:55.217677 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:48:55.218055 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:55.218067 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:55.220157 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:55.220162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:55.222466 (+ 2304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:55.222483 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:55.224730 (+ 2247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:55.224736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:55.224917 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:55.224956 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 00:48:55.225033 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 00:48:55.225147 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":638,"threads_started":1}
I20260810 00:48:55.225859 20260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:55.216280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42850 (local address 127.28.6.1:35567)
0810 00:48:55.217023 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:55.217032 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:55.217059 (+ 27us) server_negotiation.cc:407] Connection header received
0810 00:48:55.217160 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:55.217167 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:55.217418 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:55.218267 (+ 849us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:48:55.218528 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:55.219993 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:55.222654 (+ 2661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:55.224514 (+ 1860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:55.224643 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:55.225137 (+ 494us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:55.225216 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 00:48:55.225380 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":1052,"threads_started":1}
I20260810 00:48:55.225891 20256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:55.215392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58838 (local address 127.28.6.2:34761)
0810 00:48:55.217900 (+ 2508us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:55.217910 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:55.217939 (+ 29us) server_negotiation.cc:407] Connection header received
0810 00:48:55.218043 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:55.218050 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:55.218256 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:55.218462 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:48:55.219147 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:55.220648 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:55.222443 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:55.223947 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:55.225102 (+ 1155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:55.225279 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:55.225353 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 00:48:55.225517 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2347,"thread_start_us":2233,"threads_started":1}
I20260810 00:48:55.228884 19675 tablet_service.cc:1467] Processing CreateTablet for tablet 3880a1f078344e13b1d54b32a50e8bb1 (DEFAULT_TABLE table=test-table [id=975e05666f0c4e8f9f5e15037738b26f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:48:55.229105 19902 tablet_service.cc:1467] Processing CreateTablet for tablet 3880a1f078344e13b1d54b32a50e8bb1 (DEFAULT_TABLE table=test-table [id=975e05666f0c4e8f9f5e15037738b26f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:48:55.228989 20117 tablet_service.cc:1467] Processing CreateTablet for tablet 3880a1f078344e13b1d54b32a50e8bb1 (DEFAULT_TABLE table=test-table [id=975e05666f0c4e8f9f5e15037738b26f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:48:55.230345 19675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3880a1f078344e13b1d54b32a50e8bb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:48:55.230423 20117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3880a1f078344e13b1d54b32a50e8bb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:48:55.230423 19902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3880a1f078344e13b1d54b32a50e8bb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:48:55.243394 20262 tablet_bootstrap.cc:492] T 3880a1f078344e13b1d54b32a50e8bb1 P 92f2b001cfd2415dba06994af259aabb: Bootstrap starting.
I20260810 00:48:55.245366 20264 tablet_bootstrap.cc:492] T 3880a1f078344e13b1d54b32a50e8bb1 P 92c7bf0b0c8a4ace9fed6b4576ddc4ea: Bootstrap starting.
I20260810 00:48:55.246937 20269 tablet_bootstrap.cc:492] T 3880a1f078344e13b1d54b32a50e8bb1 P c2aa322985374b8092a1f4adb947ab42: Bootstrap starting.
I20260810 00:48:55.247493 20262 tablet_bootstrap.cc:654] T 3880a1f078344e13b1d54b32a50e8bb1 P 92f2b001cfd2415dba06994af259aabb: Neither blocks nor log segments found. Creating new log.
I20260810 00:48:55.248831 20264 tablet_bootstrap.cc:654] T 3880a1f078344e13b1d54b32a50e8bb1 P 92c7bf0b0c8a4ace9fed6b4576ddc4ea: Neither blocks nor log segments found. Creating new log.
I20260810 00:48:55.248996 20262 log.cc:824] T 3880a1f078344e13b1d54b32a50e8bb1 P 92f2b001cfd2415dba06994af259aabb: Log is configured to *not* fsync() on all Append() calls
I20260810 00:48:55.250176 20269 tablet_bootstrap.cc:654] T 3880a1f078344e13b1d54b32a50e8bb1 P c2aa322985374b8092a1f4adb947ab42: Neither blocks nor log segments found. Creating new log.
I20260810 00:48:55.250239 20264 log.cc:824] T 3880a1f078344e13b1d54b32a50e8bb1 P 92c7bf0b0c8a4ace9fed6b4576ddc4ea: Log is configured to *not* fsync() on all Append() calls
I20260810 00:48:55.251358 20269 log.cc:824] T 3880a1f078344e13b1d54b32a50e8bb1 P c2aa322985374b8092a1f4adb947ab42: Log is configured to *not* fsync() on all Append() calls
I20260810 00:48:55.253545 20262 tablet_bootstrap.cc:492] T 3880a1f078344e13b1d54b32a50e8bb1 P 92f2b001cfd2415dba06994af259aabb: No bootstrap required, opened a new log
I20260810 00:48:55.253546 20264 tablet_bootstrap.cc:492] T 3880a1f078344e13b1d54b32a50e8bb1 P 92c7bf0b0c8a4ace9fed6b4576ddc4ea: No bootstrap required, opened a new log
I20260810 00:48:55.253821 20264 ts_tablet_manager.cc:1397] T 3880a1f078344e13b1d54b32a50e8bb1 P 92c7bf0b0c8a4ace9fed6b4576ddc4ea: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.002s
I20260810 00:48:55.253840 20262 ts_tablet_manager.cc:1397] T 3880a1f078344e13b1d54b32a50e8bb1 P 92f2b001cfd2415dba06994af259aabb: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20260810 00:48:55.253906 20269 tablet_bootstrap.cc:492] T 3880a1f078344e13b1d54b32a50e8bb1 P c2aa322985374b8092a1f4adb947ab42: No bootstrap required, opened a new log
I20260810 00:48:55.254179 20269 ts_tablet_manager.cc:1397] T 3880a1f078344e13b1d54b32a50e8bb1 P c2aa322985374b8092a1f4adb947ab42: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260810 00:48:55.261744 20269 raft_consensus.cc:348] T 3880a1f078344e13b1d54b32a50e8bb1 P c2aa322985374b8092a1f4adb947ab42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2aa322985374b8092a1f4adb947ab42" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46009 } } peers { permanent_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34761 } } peers { permanent_uuid: "92f2b001cfd2415dba06994af259aabb" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 35567 } }
I20260810 00:48:55.262142 20269 raft_consensus.cc:374] T 3880a1f078344e13b1d54b32a50e8bb1 P c2aa322985374b8092a1f4adb947ab42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:48:55.262270 20269 raft_consensus.cc:729] T 3880a1f078344e13b1d54b32a50e8bb1 P c2aa322985374b8092a1f4adb947ab42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2aa322985374b8092a1f4adb947ab42, State: Initialized, Role: FOLLOWER
I20260810 00:48:55.262959 20269 consensus_queue.cc:260] T 3880a1f078344e13b1d54b32a50e8bb1 P c2aa322985374b8092a1f4adb947ab42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2aa322985374b8092a1f4adb947ab42" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46009 } } peers { permanent_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34761 } } peers { permanent_uuid: "92f2b001cfd2415dba06994af259aabb" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 35567 } }
I20260810 00:48:55.264053 20227 heartbeater.cc:503] Master 127.28.6.62:44053 was elected leader, sending a full tablet report...
I20260810 00:48:55.264781 20269 ts_tablet_manager.cc:1428] T 3880a1f078344e13b1d54b32a50e8bb1 P c2aa322985374b8092a1f4adb947ab42: Time spent starting tablet: real 0.010s user 0.011s sys 0.001s
I20260810 00:48:55.264690 20264 raft_consensus.cc:348] T 3880a1f078344e13b1d54b32a50e8bb1 P 92c7bf0b0c8a4ace9fed6b4576ddc4ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2aa322985374b8092a1f4adb947ab42" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46009 } } peers { permanent_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34761 } } peers { permanent_uuid: "92f2b001cfd2415dba06994af259aabb" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 35567 } }
I20260810 00:48:55.264690 20262 raft_consensus.cc:348] T 3880a1f078344e13b1d54b32a50e8bb1 P 92f2b001cfd2415dba06994af259aabb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2aa322985374b8092a1f4adb947ab42" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46009 } } peers { permanent_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34761 } } peers { permanent_uuid: "92f2b001cfd2415dba06994af259aabb" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 35567 } }
I20260810 00:48:55.265115 20262 raft_consensus.cc:374] T 3880a1f078344e13b1d54b32a50e8bb1 P 92f2b001cfd2415dba06994af259aabb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:48:55.265115 20264 raft_consensus.cc:374] T 3880a1f078344e13b1d54b32a50e8bb1 P 92c7bf0b0c8a4ace9fed6b4576ddc4ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:48:55.265224 20264 raft_consensus.cc:729] T 3880a1f078344e13b1d54b32a50e8bb1 P 92c7bf0b0c8a4ace9fed6b4576ddc4ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92c7bf0b0c8a4ace9fed6b4576ddc4ea, State: Initialized, Role: FOLLOWER
I20260810 00:48:55.265250 20262 raft_consensus.cc:729] T 3880a1f078344e13b1d54b32a50e8bb1 P 92f2b001cfd2415dba06994af259aabb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92f2b001cfd2415dba06994af259aabb, State: Initialized, Role: FOLLOWER
I20260810 00:48:55.265818 20264 consensus_queue.cc:260] T 3880a1f078344e13b1d54b32a50e8bb1 P 92c7bf0b0c8a4ace9fed6b4576ddc4ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2aa322985374b8092a1f4adb947ab42" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46009 } } peers { permanent_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34761 } } peers { permanent_uuid: "92f2b001cfd2415dba06994af259aabb" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 35567 } }
I20260810 00:48:55.265906 20262 consensus_queue.cc:260] T 3880a1f078344e13b1d54b32a50e8bb1 P 92f2b001cfd2415dba06994af259aabb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2aa322985374b8092a1f4adb947ab42" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46009 } } peers { permanent_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34761 } } peers { permanent_uuid: "92f2b001cfd2415dba06994af259aabb" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 35567 } }
I20260810 00:48:55.266959 20010 heartbeater.cc:503] Master 127.28.6.62:44053 was elected leader, sending a full tablet report...
I20260810 00:48:55.267870 20264 ts_tablet_manager.cc:1428] T 3880a1f078344e13b1d54b32a50e8bb1 P 92c7bf0b0c8a4ace9fed6b4576ddc4ea: Time spent starting tablet: real 0.014s user 0.012s sys 0.003s
I20260810 00:48:55.269992 20262 ts_tablet_manager.cc:1428] T 3880a1f078344e13b1d54b32a50e8bb1 P 92f2b001cfd2415dba06994af259aabb: Time spent starting tablet: real 0.016s user 0.008s sys 0.009s
I20260810 00:48:55.327701 20277 raft_consensus.cc:482] T 3880a1f078344e13b1d54b32a50e8bb1 P 92f2b001cfd2415dba06994af259aabb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:48:55.328064 20277 raft_consensus.cc:504] T 3880a1f078344e13b1d54b32a50e8bb1 P 92f2b001cfd2415dba06994af259aabb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2aa322985374b8092a1f4adb947ab42" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46009 } } peers { permanent_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34761 } } peers { permanent_uuid: "92f2b001cfd2415dba06994af259aabb" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 35567 } }
I20260810 00:48:55.329730 20277 leader_election.cc:288] T 3880a1f078344e13b1d54b32a50e8bb1 P 92f2b001cfd2415dba06994af259aabb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2aa322985374b8092a1f4adb947ab42 (127.28.6.3:46009), 92c7bf0b0c8a4ace9fed6b4576ddc4ea (127.28.6.2:34761)
W20260810 00:48:55.331215 19789 tablet.cc:2369] T 3880a1f078344e13b1d54b32a50e8bb1 P 92f2b001cfd2415dba06994af259aabb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:48:55.337486 20252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:55.330191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:49867 (local address 127.28.6.3:46009)
0810 00:48:55.330384 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:55.330389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:55.330970 (+ 581us) server_negotiation.cc:407] Connection header received
0810 00:48:55.331352 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:55.331360 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:55.331467 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:55.331546 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:48:55.332387 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:55.333464 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:55.335528 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:55.336522 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:55.336618 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:55.336696 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:55.337100 (+ 404us) server_negotiation.cc:299] Negotiation successful
0810 00:48:55.337190 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 00:48:55.337647 20278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:55.330081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:46009 (local address 127.28.6.1:49867)
0810 00:48:55.330841 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:55.330860 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:55.331102 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:55.331802 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:55.331806 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:55.331836 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:48:55.332242 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:55.332252 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:55.333603 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:55.333609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:55.335368 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:55.335380 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:55.336751 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:55.336758 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:55.336933 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:55.336969 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 00:48:55.337074 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 00:48:55.337205 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"mutex_wait_us":1,"thread_start_us":401,"threads_started":1}
I20260810 00:48:55.338215 20256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:55.331253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:58079 (local address 127.28.6.2:34761)
0810 00:48:55.331478 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:55.331484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:55.332228 (+ 744us) server_negotiation.cc:407] Connection header received
0810 00:48:55.332478 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:55.332482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:55.332539 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:55.332606 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:48:55.333221 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:55.334173 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:55.336145 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:55.337142 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:55.337277 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:55.337382 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:55.337840 (+ 458us) server_negotiation.cc:299] Negotiation successful
0810 00:48:55.337970 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 00:48:55.338265 20279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:55.330955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:34761 (local address 127.28.6.1:58079)
0810 00:48:55.332088 (+ 1133us) negotiation.cc:107] Waiting for socket to connect
0810 00:48:55.332116 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 00:48:55.332318 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:48:55.332667 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:48:55.332672 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:48:55.332699 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:48:55.333069 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:55.333081 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:55.334328 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:55.334331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:55.335988 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:48:55.336003 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:48:55.337395 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:48:55.337402 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:48:55.337602 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:55.337645 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 00:48:55.337809 (+ 164us) client_negotiation.cc:240] Negotiation successful
0810 00:48:55.337934 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":981,"thread_start_us":774,"threads_started":1}
I20260810 00:48:55.339071 20150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3880a1f078344e13b1d54b32a50e8bb1" candidate_uuid: "92f2b001cfd2415dba06994af259aabb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2aa322985374b8092a1f4adb947ab42" is_pre_election: true
I20260810 00:48:55.339290 19935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3880a1f078344e13b1d54b32a50e8bb1" candidate_uuid: "92f2b001cfd2415dba06994af259aabb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea" is_pre_election: true
I20260810 00:48:55.339620 20150 raft_consensus.cc:2454] T 3880a1f078344e13b1d54b32a50e8bb1 P c2aa322985374b8092a1f4adb947ab42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92f2b001cfd2415dba06994af259aabb in term 0.
I20260810 00:48:55.339812 19935 raft_consensus.cc:2454] T 3880a1f078344e13b1d54b32a50e8bb1 P 92c7bf0b0c8a4ace9fed6b4576ddc4ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92f2b001cfd2415dba06994af259aabb in term 0.
I20260810 00:48:55.340394 19602 leader_election.cc:302] T 3880a1f078344e13b1d54b32a50e8bb1 P 92f2b001cfd2415dba06994af259aabb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92f2b001cfd2415dba06994af259aabb, c2aa322985374b8092a1f4adb947ab42; no voters:
I20260810 00:48:55.340818 20277 raft_consensus.cc:2781] T 3880a1f078344e13b1d54b32a50e8bb1 P 92f2b001cfd2415dba06994af259aabb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 00:48:55.340999 20277 raft_consensus.cc:482] T 3880a1f078344e13b1d54b32a50e8bb1 P 92f2b001cfd2415dba06994af259aabb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 00:48:55.341135 20277 raft_consensus.cc:3037] T 3880a1f078344e13b1d54b32a50e8bb1 P 92f2b001cfd2415dba06994af259aabb [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:48:55.343433 20277 raft_consensus.cc:504] T 3880a1f078344e13b1d54b32a50e8bb1 P 92f2b001cfd2415dba06994af259aabb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2aa322985374b8092a1f4adb947ab42" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46009 } } peers { permanent_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34761 } } peers { permanent_uuid: "92f2b001cfd2415dba06994af259aabb" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 35567 } }
I20260810 00:48:55.344007 20277 leader_election.cc:288] T 3880a1f078344e13b1d54b32a50e8bb1 P 92f2b001cfd2415dba06994af259aabb [CANDIDATE]: Term 1 election: Requested vote from peers c2aa322985374b8092a1f4adb947ab42 (127.28.6.3:46009), 92c7bf0b0c8a4ace9fed6b4576ddc4ea (127.28.6.2:34761)
I20260810 00:48:55.344410 20150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3880a1f078344e13b1d54b32a50e8bb1" candidate_uuid: "92f2b001cfd2415dba06994af259aabb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2aa322985374b8092a1f4adb947ab42"
I20260810 00:48:55.344524 19935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3880a1f078344e13b1d54b32a50e8bb1" candidate_uuid: "92f2b001cfd2415dba06994af259aabb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea"
I20260810 00:48:55.344601 20150 raft_consensus.cc:3037] T 3880a1f078344e13b1d54b32a50e8bb1 P c2aa322985374b8092a1f4adb947ab42 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:48:55.344743 19935 raft_consensus.cc:3037] T 3880a1f078344e13b1d54b32a50e8bb1 P 92c7bf0b0c8a4ace9fed6b4576ddc4ea [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:48:55.346791 20150 raft_consensus.cc:2454] T 3880a1f078344e13b1d54b32a50e8bb1 P c2aa322985374b8092a1f4adb947ab42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92f2b001cfd2415dba06994af259aabb in term 1.
I20260810 00:48:55.346956 19935 raft_consensus.cc:2454] T 3880a1f078344e13b1d54b32a50e8bb1 P 92c7bf0b0c8a4ace9fed6b4576ddc4ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92f2b001cfd2415dba06994af259aabb in term 1.
I20260810 00:48:55.347208 19602 leader_election.cc:302] T 3880a1f078344e13b1d54b32a50e8bb1 P 92f2b001cfd2415dba06994af259aabb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92f2b001cfd2415dba06994af259aabb, c2aa322985374b8092a1f4adb947ab42; no voters:
I20260810 00:48:55.347491 20277 raft_consensus.cc:2781] T 3880a1f078344e13b1d54b32a50e8bb1 P 92f2b001cfd2415dba06994af259aabb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:48:55.347924 20277 raft_consensus.cc:686] T 3880a1f078344e13b1d54b32a50e8bb1 P 92f2b001cfd2415dba06994af259aabb [term 1 LEADER]: Becoming Leader. State: Replica: 92f2b001cfd2415dba06994af259aabb, State: Running, Role: LEADER
I20260810 00:48:55.348378 20277 consensus_queue.cc:237] T 3880a1f078344e13b1d54b32a50e8bb1 P 92f2b001cfd2415dba06994af259aabb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2aa322985374b8092a1f4adb947ab42" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46009 } } peers { permanent_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34761 } } peers { permanent_uuid: "92f2b001cfd2415dba06994af259aabb" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 35567 } }
I20260810 00:48:55.353111 19473 catalog_manager.cc:5495] T 3880a1f078344e13b1d54b32a50e8bb1 P 92f2b001cfd2415dba06994af259aabb reported cstate change: term changed from 0 to 1, leader changed from <none> to 92f2b001cfd2415dba06994af259aabb (127.28.6.1). New cstate: current_term: 1 leader_uuid: "92f2b001cfd2415dba06994af259aabb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2aa322985374b8092a1f4adb947ab42" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92f2b001cfd2415dba06994af259aabb" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 35567 } health_report { overall_health: HEALTHY } } }
W20260810 00:48:55.367578 20011 tablet.cc:2369] T 3880a1f078344e13b1d54b32a50e8bb1 P 92c7bf0b0c8a4ace9fed6b4576ddc4ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:48:55.383384 20260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:55.377643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42860 (local address 127.28.6.1:35567)
0810 00:48:55.377836 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:55.377842 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:55.377939 (+ 97us) server_negotiation.cc:407] Connection header received
0810 00:48:55.378165 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:55.378170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:55.378274 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:55.378419 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:48:55.378800 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:55.379606 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:55.381173 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:55.381654 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:55.381752 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:55.382264 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:48:55.382761 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:48:55.382867 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:55.383038 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 00:48:55.383120 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260810 00:48:55.390056 20230 tablet.cc:2369] T 3880a1f078344e13b1d54b32a50e8bb1 P c2aa322985374b8092a1f4adb947ab42: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:48:55.391492 19935 raft_consensus.cc:1261] T 3880a1f078344e13b1d54b32a50e8bb1 P 92c7bf0b0c8a4ace9fed6b4576ddc4ea [term 1 FOLLOWER]: Refusing update from remote peer 92f2b001cfd2415dba06994af259aabb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:48:55.391533 20150 raft_consensus.cc:1261] T 3880a1f078344e13b1d54b32a50e8bb1 P c2aa322985374b8092a1f4adb947ab42 [term 1 FOLLOWER]: Refusing update from remote peer 92f2b001cfd2415dba06994af259aabb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:48:55.392402 20277 consensus_queue.cc:1035] T 3880a1f078344e13b1d54b32a50e8bb1 P 92f2b001cfd2415dba06994af259aabb [LEADER]: Connected to new peer: Peer: permanent_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:48:55.392764 20280 consensus_queue.cc:1035] T 3880a1f078344e13b1d54b32a50e8bb1 P 92f2b001cfd2415dba06994af259aabb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2aa322985374b8092a1f4adb947ab42" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:48:55.403944 20284 mvcc.cc:205] Tried to move back new op lower bound from 7316778743352909824 to 7316778743190765568. Current Snapshot: MvccSnapshot[applied={T|T < 7316778743352909824}]
I20260810 00:48:55.405710 20286 mvcc.cc:205] Tried to move back new op lower bound from 7316778743352909824 to 7316778743190765568. Current Snapshot: MvccSnapshot[applied={T|T < 7316778743352909824}]
I20260810 00:48:55.410455 20287 mvcc.cc:205] Tried to move back new op lower bound from 7316778743352909824 to 7316778743190765568. Current Snapshot: MvccSnapshot[applied={T|T < 7316778743352909824}]
I20260810 00:48:55.420614 19473 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:48:55.420917 19473 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:48:55.424535 19473 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb: Sending DeleteTablet for 3 replicas of tablet 3880a1f078344e13b1d54b32a50e8bb1
I20260810 00:48:55.425581 20117 tablet_service.cc:1514] Processing DeleteTablet for tablet 3880a1f078344e13b1d54b32a50e8bb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:48:55 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:60086
I20260810 00:48:55.425954 19675 tablet_service.cc:1514] Processing DeleteTablet for tablet 3880a1f078344e13b1d54b32a50e8bb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:48:55 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:42850
I20260810 00:48:55.426060 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 19570
I20260810 00:48:55.426312 19902 tablet_service.cc:1514] Processing DeleteTablet for tablet 3880a1f078344e13b1d54b32a50e8bb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:48:55 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:58838
I20260810 00:48:55.437147 20306 tablet_replica.cc:330] T 3880a1f078344e13b1d54b32a50e8bb1 P 92c7bf0b0c8a4ace9fed6b4576ddc4ea: stopping tablet replica
I20260810 00:48:55.437561 20306 raft_consensus.cc:2229] T 3880a1f078344e13b1d54b32a50e8bb1 P 92c7bf0b0c8a4ace9fed6b4576ddc4ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:48:55.437858 20301 tablet_replica.cc:330] T 3880a1f078344e13b1d54b32a50e8bb1 P c2aa322985374b8092a1f4adb947ab42: stopping tablet replica
I20260810 00:48:55.438251 20301 raft_consensus.cc:2229] T 3880a1f078344e13b1d54b32a50e8bb1 P c2aa322985374b8092a1f4adb947ab42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:48:55.438750 20301 raft_consensus.cc:2258] T 3880a1f078344e13b1d54b32a50e8bb1 P c2aa322985374b8092a1f4adb947ab42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:48:55.439412 20303 tablet_replica.cc:330] T 3880a1f078344e13b1d54b32a50e8bb1 P 92f2b001cfd2415dba06994af259aabb: stopping tablet replica
I20260810 00:48:55.439457 20306 raft_consensus.cc:2258] T 3880a1f078344e13b1d54b32a50e8bb1 P 92c7bf0b0c8a4ace9fed6b4576ddc4ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:48:55.439801 20303 raft_consensus.cc:2229] T 3880a1f078344e13b1d54b32a50e8bb1 P 92f2b001cfd2415dba06994af259aabb [term 1 LEADER]: Raft consensus shutting down.
I20260810 00:48:55.440491 20303 raft_consensus.cc:2258] T 3880a1f078344e13b1d54b32a50e8bb1 P 92f2b001cfd2415dba06994af259aabb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:48:55.441615 20306 ts_tablet_manager.cc:1905] T 3880a1f078344e13b1d54b32a50e8bb1 P 92c7bf0b0c8a4ace9fed6b4576ddc4ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:48:55.443267 20301 ts_tablet_manager.cc:1905] T 3880a1f078344e13b1d54b32a50e8bb1 P c2aa322985374b8092a1f4adb947ab42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:48:55.450394 20301 ts_tablet_manager.cc:1918] T 3880a1f078344e13b1d54b32a50e8bb1 P c2aa322985374b8092a1f4adb947ab42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:48:55.450555 20301 log.cc:1196] T 3880a1f078344e13b1d54b32a50e8bb1 P c2aa322985374b8092a1f4adb947ab42: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/wal/wals/3880a1f078344e13b1d54b32a50e8bb1
I20260810 00:48:55.451094 20301 ts_tablet_manager.cc:1939] T 3880a1f078344e13b1d54b32a50e8bb1 P c2aa322985374b8092a1f4adb947ab42: Deleting consensus metadata
I20260810 00:48:55.452351 20303 ts_tablet_manager.cc:1905] T 3880a1f078344e13b1d54b32a50e8bb1 P 92f2b001cfd2415dba06994af259aabb: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 10 00:48:55 dist-test-slave-1nl6 krb5kdc[19398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.1@KRBTEST.COM
I20260810 00:48:55.453042 19452 catalog_manager.cc:4841] TS c2aa322985374b8092a1f4adb947ab42 (127.28.6.3:46009): tablet 3880a1f078344e13b1d54b32a50e8bb1 (table test-table [id=975e05666f0c4e8f9f5e15037738b26f]) successfully deleted
I20260810 00:48:55.460130 20306 ts_tablet_manager.cc:1918] T 3880a1f078344e13b1d54b32a50e8bb1 P 92c7bf0b0c8a4ace9fed6b4576ddc4ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:48:55.460201 20303 ts_tablet_manager.cc:1918] T 3880a1f078344e13b1d54b32a50e8bb1 P 92f2b001cfd2415dba06994af259aabb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:48:55.460315 20306 log.cc:1196] T 3880a1f078344e13b1d54b32a50e8bb1 P 92c7bf0b0c8a4ace9fed6b4576ddc4ea: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/wal/wals/3880a1f078344e13b1d54b32a50e8bb1
I20260810 00:48:55.460386 20303 log.cc:1196] T 3880a1f078344e13b1d54b32a50e8bb1 P 92f2b001cfd2415dba06994af259aabb: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/wal/wals/3880a1f078344e13b1d54b32a50e8bb1
I20260810 00:48:55.460873 20306 ts_tablet_manager.cc:1939] T 3880a1f078344e13b1d54b32a50e8bb1 P 92c7bf0b0c8a4ace9fed6b4576ddc4ea: Deleting consensus metadata
I20260810 00:48:55.460975 20303 ts_tablet_manager.cc:1939] T 3880a1f078344e13b1d54b32a50e8bb1 P 92f2b001cfd2415dba06994af259aabb: Deleting consensus metadata
I20260810 00:48:55.462656 19452 catalog_manager.cc:4841] TS 92f2b001cfd2415dba06994af259aabb (127.28.6.1:35567): tablet 3880a1f078344e13b1d54b32a50e8bb1 (table test-table [id=975e05666f0c4e8f9f5e15037738b26f]) successfully deleted
I20260810 00:48:55.462977 19452 catalog_manager.cc:4841] TS 92c7bf0b0c8a4ace9fed6b4576ddc4ea (127.28.6.2:34761): tablet 3880a1f078344e13b1d54b32a50e8bb1 (table test-table [id=975e05666f0c4e8f9f5e15037738b26f]) successfully deleted
I20260810 00:48:55.463580 20260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:55.427276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42870 (local address 127.28.6.1:35567)
0810 00:48:55.427483 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:55.427492 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:55.441204 (+ 13712us) server_negotiation.cc:407] Connection header received
0810 00:48:55.441412 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:55.441417 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:55.441498 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:55.441593 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:55.443785 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:55.444920 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:55.447394 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:55.448085 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:55.448203 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:55.454464 (+ 6261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:55.454486 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:55.454504 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:55.454547 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:55.458972 (+ 4425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:55.459935 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:55.459946 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:55.459952 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:55.460148 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:55.460909 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:55.460917 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:55.460921 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:55.461496 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:55.461832 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:55.462888 (+ 1056us) server_negotiation.cc:299] Negotiation successful
0810 00:48:55.463214 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 00:48:55.464603 19776 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:48:55.563938 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 19570
I20260810 00:48:55.590279 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 19799
Aug 10 00:48:55 dist-test-slave-1nl6 krb5kdc[19398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.2@KRBTEST.COM
I20260810 00:48:55.605247 20256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:55.591215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58846 (local address 127.28.6.2:34761)
0810 00:48:55.591412 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:55.591417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:55.591436 (+ 19us) server_negotiation.cc:407] Connection header received
0810 00:48:55.591582 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:55.591586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:55.591659 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:55.591753 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:55.593165 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:55.593895 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:55.595327 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:55.595710 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:55.595805 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:55.599783 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:55.599795 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:55.599811 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:55.599841 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:55.602692 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:55.603454 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:55.603461 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:55.603464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:55.603582 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:55.603911 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:55.603914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:55.603916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:55.604255 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:55.604471 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:55.604719 (+ 248us) server_negotiation.cc:299] Negotiation successful
0810 00:48:55.604878 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 00:48:55.605964 20002 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:48:55.695802 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 19799
I20260810 00:48:55.723938 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 20019
Aug 10 00:48:55 dist-test-slave-1nl6 krb5kdc[19398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.3@KRBTEST.COM
I20260810 00:48:55.738432 20252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:48:55.724754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60094 (local address 127.28.6.3:46009)
0810 00:48:55.724924 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 00:48:55.724929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:48:55.724947 (+ 18us) server_negotiation.cc:407] Connection header received
0810 00:48:55.725084 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:48:55.725090 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:48:55.725192 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:48:55.725308 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:48:55.726644 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:55.727445 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:55.728754 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:48:55.729136 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:48:55.729225 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:48:55.733389 (+ 4164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:48:55.733406 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:48:55.733426 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:48:55.733462 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:48:55.736054 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:55.736710 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:55.736719 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:55.736721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:55.736848 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:48:55.737157 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:48:55.737160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:48:55.737162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:48:55.737520 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:48:55.737734 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0810 00:48:55.738013 (+ 279us) server_negotiation.cc:299] Negotiation successful
0810 00:48:55.738173 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 00:48:55.739149 20220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:48:55.828183 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 20019
I20260810 00:48:55.855329 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 19419
I20260810 00:48:55.856596 19524 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:48:55.965000 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 19419
I20260810 00:49:15.995820 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:44053
--webserver_interface=127.28.6.62
--webserver_port=36879
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:39585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:44053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:49:16.340706 20315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:49:16.341005 20315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:49:16.341168 20315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:49:16.348376 20315 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 00:49:16.348503 20315 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 00:49:16.348606 20315 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 00:49:16.348702 20315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:49:16.348794 20315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:49:16.348906 20315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:49:16.348989 20315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:49:16.349095 20315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:49:16.358677 20315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:39585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:44053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:44053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=36879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:49:16.359964 20315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:49:16.361312 20315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:49:16.386432 20328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:49:16.386339 20333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:49:16.386375 20329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:49:16.387279 20315 server_base.cc:1034] running on GCE node
I20260810 00:49:16.388116 20315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:49:16.389611 20315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:49:16.390836 20315 hybrid_clock.cc:648] HybridClock initialized: now 1786322956390775 us; error 29 us; skew 500 ppm
I20260810 00:49:16.391258 20315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:49:16 dist-test-slave-1nl6 krb5kdc[19398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322956, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:49:16.400681 20315 init.cc:377] Logged in from keytab as kudu/127.28.6.62@KRBTEST.COM (short username kudu)
I20260810 00:49:16.402410 20315 webserver.cc:460] Webserver started at http://127.28.6.62:36879/ using document root <none> and password file <none>
I20260810 00:49:16.403301 20315 fs_manager.cc:359] Metadata directory not provided
I20260810 00:49:16.403458 20315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:49:16.409540 20315 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.005s
I20260810 00:49:16.412536 20348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:49:16.413831 20315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260810 00:49:16.414028 20315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "491fe8c474e84bd9b275042408d6adbb"
format_stamp: "Formatted at 2026-08-10 00:48:53 on dist-test-slave-1nl6"
server_key: "9782b27bfcedfd6c8521961244e154bf"
server_key_iv: "677b1b6c806eb2c5e6812568cbcc18f1"
server_key_version: "encryptionkey@0"
I20260810 00:49:16.415122 20315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:49:16.430590 20315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:49:16.434523 20315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:49:16.434813 20315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:49:16.454047 20315 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:44053
I20260810 00:49:16.454098 20441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:44053 every 8 connection(s)
I20260810 00:49:16.455725 20315 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:49:16.462708 20442 sys_catalog.cc:263] Verifying existing consensus state
I20260810 00:49:16.465355 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 20315
I20260810 00:49:16.466427 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.1:35567
--local_ip_for_outbound_sockets=127.28.6.1
--tserver_master_addrs=127.28.6.62:44053
--webserver_port=46097
--webserver_interface=127.28.6.1
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:39585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 00:49:16.466490 20442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb: Bootstrap starting.
I20260810 00:49:16.489804 20442 log.cc:824] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb: Log is configured to *not* fsync() on all Append() calls
I20260810 00:49:16.501771 20442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 00:49:16.502322 20442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb: Bootstrap complete.
I20260810 00:49:16.511580 20442 raft_consensus.cc:348] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491fe8c474e84bd9b275042408d6adbb" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 44053 } }
I20260810 00:49:16.512408 20442 raft_consensus.cc:729] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491fe8c474e84bd9b275042408d6adbb, State: Initialized, Role: FOLLOWER
I20260810 00:49:16.513016 20442 consensus_queue.cc:260] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491fe8c474e84bd9b275042408d6adbb" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 44053 } }
I20260810 00:49:16.513258 20442 raft_consensus.cc:388] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:49:16.513404 20442 raft_consensus.cc:482] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:49:16.513583 20442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [term 1 FOLLOWER]: Advancing to term 2
I20260810 00:49:16.517374 20442 raft_consensus.cc:504] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491fe8c474e84bd9b275042408d6adbb" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 44053 } }
I20260810 00:49:16.517872 20442 leader_election.cc:302] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 491fe8c474e84bd9b275042408d6adbb; no voters:
I20260810 00:49:16.518548 20442 leader_election.cc:288] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 00:49:16.518694 20452 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [term 2 FOLLOWER]: Leader election won for term 2
I20260810 00:49:16.521150 20442 sys_catalog.cc:564] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:49:16.525759 20452 raft_consensus.cc:686] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [term 2 LEADER]: Becoming Leader. State: Replica: 491fe8c474e84bd9b275042408d6adbb, State: Running, Role: LEADER
I20260810 00:49:16.526422 20452 consensus_queue.cc:237] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491fe8c474e84bd9b275042408d6adbb" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 44053 } }
I20260810 00:49:16.529249 20454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "491fe8c474e84bd9b275042408d6adbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491fe8c474e84bd9b275042408d6adbb" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 44053 } } }
I20260810 00:49:16.529683 20454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [sys.catalog]: This master's current role is: LEADER
I20260810 00:49:16.536724 20455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 491fe8c474e84bd9b275042408d6adbb. Latest consensus state: current_term: 2 leader_uuid: "491fe8c474e84bd9b275042408d6adbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491fe8c474e84bd9b275042408d6adbb" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 44053 } } }
I20260810 00:49:16.537113 20455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [sys.catalog]: This master's current role is: LEADER
I20260810 00:49:16.537326 20465 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:49:16.545583 20465 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:49:16.547021 20465 catalog_manager.cc:1259] Loaded cluster ID: 8e4bca419786490cbc821bfe13a74ebc
I20260810 00:49:16.547168 20465 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:49:16.551349 20465 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:49:16.553002 20465 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb: Loaded TSK: 0
I20260810 00:49:16.590242 20465 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb: Generated new TSK 1
I20260810 00:49:16.590642 20465 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 00:49:16.833946 20447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:49:16.834276 20447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:49:16.834446 20447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:49:16.842835 20447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:49:16.843001 20447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:49:16.843171 20447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.1
I20260810 00:49:16.852715 20447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:39585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.1:35567
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.6.1
--webserver_port=46097
--tserver_master_addrs=127.28.6.62:44053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.1
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:49:16.854022 20447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:49:16.855449 20447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:49:16.870332 20492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:49:16.870369 20490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:49:16.871708 20496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:49:16.872220 20447 server_base.cc:1034] running on GCE node
I20260810 00:49:16.872843 20447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:49:16.873840 20447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:49:16.875051 20447 hybrid_clock.cc:648] HybridClock initialized: now 1786322956875007 us; error 38 us; skew 500 ppm
I20260810 00:49:16.875371 20447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:49:16 dist-test-slave-1nl6 krb5kdc[19398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322956, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:49:16.878772 20447 init.cc:377] Logged in from keytab as kudu/127.28.6.1@KRBTEST.COM (short username kudu)
I20260810 00:49:16.880437 20447 webserver.cc:460] Webserver started at http://127.28.6.1:46097/ using document root <none> and password file <none>
I20260810 00:49:16.881119 20447 fs_manager.cc:359] Metadata directory not provided
I20260810 00:49:16.881276 20447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:49:16.886535 20447 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 00:49:16.889281 20508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:49:16.890617 20447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 00:49:16.890789 20447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/wal
uuid: "92f2b001cfd2415dba06994af259aabb"
format_stamp: "Formatted at 2026-08-10 00:48:53 on dist-test-slave-1nl6"
server_key: "d41800cbef261e62d0c2da089016b595"
server_key_iv: "801c02d3a8d6bf5d64dcc54e4375f125"
server_key_version: "encryptionkey@0"
I20260810 00:49:16.891769 20447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:49:16.916136 20447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:49:16.920526 20447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:49:16.920786 20447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:49:16.921767 20447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:49:16.923192 20447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:49:16.923326 20447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:49:16.923440 20447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:49:16.923519 20447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:49:16.957010 20447 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.1:35567
I20260810 00:49:16.957208 20705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.1:35567 every 8 connection(s)
I20260810 00:49:16.958428 20447 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
I20260810 00:49:16.967872 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 20447
I20260810 00:49:16.968667 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.2:34761
--local_ip_for_outbound_sockets=127.28.6.2
--tserver_master_addrs=127.28.6.62:44053
--webserver_port=41307
--webserver_interface=127.28.6.2
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:39585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 00:49:16 dist-test-slave-1nl6 krb5kdc[19398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322956, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:49:16.984805 20708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:16.961386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:35499 (local address 127.28.6.62:44053)
0810 00:49:16.961946 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0810 00:49:16.961957 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 00:49:16.962774 (+ 817us) server_negotiation.cc:407] Connection header received
0810 00:49:16.965046 (+ 2272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:49:16.965058 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:49:16.965357 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:49:16.965655 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:49:16.967197 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:16.969222 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:16.970927 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:16.971859 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:16.972082 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:16.976010 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:49:16.976023 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:49:16.976044 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:49:16.976100 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:49:16.980427 (+ 4327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:49:16.981383 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:49:16.981389 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:49:16.981395 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:49:16.981548 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:49:16.982046 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:49:16.982050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:49:16.982051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:49:16.982492 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:49:16.982644 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 00:49:16.983592 (+ 948us) server_negotiation.cc:299] Negotiation successful
0810 00:49:16.983890 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":250,"threads_started":1}
I20260810 00:49:16.985479 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:16.961612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.1:35499)
0810 00:49:16.962582 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:16.962641 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 00:49:16.964551 (+ 1910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:49:16.965825 (+ 1274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:49:16.965832 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:49:16.966365 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:49:16.967007 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:49:16.967029 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:49:16.969441 (+ 2412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:49:16.969446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:49:16.970645 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:49:16.970658 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:49:16.972111 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:49:16.972120 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:49:16.972405 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:16.973075 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:49:16.973101 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:49:16.975681 (+ 2580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:49:16.980581 (+ 4900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:49:16.980591 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:49:16.980607 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:49:16.981230 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:49:16.981659 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:49:16.981666 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:49:16.981672 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:49:16.981932 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:49:16.982630 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:49:16.982641 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:49:16.983064 (+ 423us) client_negotiation.cc:769] Sending connection context
0810 00:49:16.983384 (+ 320us) client_negotiation.cc:240] Negotiation successful
0810 00:49:16.983923 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":430,"threads_started":1}
I20260810 00:49:16.988394 20706 heartbeater.cc:347] Connected to a master server at 127.28.6.62:44053
I20260810 00:49:16.988683 20706 heartbeater.cc:464] Registering TS with master...
I20260810 00:49:16.989373 20706 heartbeater.cc:511] Master 127.28.6.62:44053 requested a full tablet report, sending...
I20260810 00:49:16.991663 20382 ts_manager.cc:194] Registered new tserver with Master: 92f2b001cfd2415dba06994af259aabb (127.28.6.1:35567)
I20260810 00:49:16.993105 20382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.1@KRBTEST.COM'} at 127.28.6.1:35499
W20260810 00:49:17.316275 20710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:49:17.316639 20710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:49:17.316820 20710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:49:17.325376 20710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:49:17.325496 20710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:49:17.325675 20710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.2
I20260810 00:49:17.335883 20710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:39585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.2:34761
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.6.2
--webserver_port=41307
--tserver_master_addrs=127.28.6.62:44053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.2
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:49:17.337224 20710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:49:17.338595 20710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:49:17.352115 20722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:49:17.352069 20727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:49:17.352200 20724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:49:17.352960 20710 server_base.cc:1034] running on GCE node
I20260810 00:49:17.353582 20710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:49:17.354521 20710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:49:17.355731 20710 hybrid_clock.cc:648] HybridClock initialized: now 1786322957355673 us; error 58 us; skew 500 ppm
I20260810 00:49:17.356058 20710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:49:17 dist-test-slave-1nl6 krb5kdc[19398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322957, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:49:17.359382 20710 init.cc:377] Logged in from keytab as kudu/127.28.6.2@KRBTEST.COM (short username kudu)
I20260810 00:49:17.360877 20710 webserver.cc:460] Webserver started at http://127.28.6.2:41307/ using document root <none> and password file <none>
I20260810 00:49:17.361573 20710 fs_manager.cc:359] Metadata directory not provided
I20260810 00:49:17.361729 20710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:49:17.367117 20710 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 00:49:17.369796 20740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:49:17.371140 20710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 00:49:17.371321 20710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/wal
uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea"
format_stamp: "Formatted at 2026-08-10 00:48:54 on dist-test-slave-1nl6"
server_key: "4930f94ecea10c721337ac33af729447"
server_key_iv: "e875c1dc1d9b0dea9f7e634fd9211dd5"
server_key_version: "encryptionkey@0"
I20260810 00:49:17.372331 20710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:49:17.386526 20710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:49:17.390439 20710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:49:17.390697 20710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:49:17.391685 20710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:49:17.392998 20710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:49:17.393137 20710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:49:17.393261 20710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:49:17.393363 20710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:49:17.428179 20710 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.2:34761
I20260810 00:49:17.428238 20922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.2:34761 every 8 connection(s)
I20260810 00:49:17.429630 20710 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
I20260810 00:49:17.430751 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 20710
I20260810 00:49:17.431545 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.3:46009
--local_ip_for_outbound_sockets=127.28.6.3
--tserver_master_addrs=127.28.6.62:44053
--webserver_port=39153
--webserver_interface=127.28.6.3
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:39585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 00:49:17 dist-test-slave-1nl6 krb5kdc[19398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322957, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:49:17.453294 20708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:17.432484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:41997 (local address 127.28.6.62:44053)
0810 00:49:17.432732 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0810 00:49:17.432741 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:49:17.434025 (+ 1284us) server_negotiation.cc:407] Connection header received
0810 00:49:17.437031 (+ 3006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:49:17.437036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:49:17.437104 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:49:17.437195 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:49:17.439342 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:17.440054 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:17.442555 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:17.443076 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:17.443179 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:17.447417 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:49:17.447422 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:49:17.447425 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:49:17.447470 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:49:17.449943 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:49:17.450959 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:49:17.450963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:49:17.450966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:49:17.451083 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:49:17.451579 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:49:17.451582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:49:17.451584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:49:17.451840 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:49:17.451951 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 00:49:17.452819 (+ 868us) server_negotiation.cc:299] Negotiation successful
0810 00:49:17.452990 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 00:49:17.454869 20926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:17.432989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.2:41997)
0810 00:49:17.433831 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:17.433889 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 00:49:17.436711 (+ 2822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:49:17.437461 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:49:17.437475 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:49:17.438259 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:49:17.439149 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:49:17.439178 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:49:17.440233 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:49:17.440241 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:49:17.442388 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:49:17.442407 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:49:17.443304 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:49:17.443314 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:49:17.443667 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:17.444425 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:49:17.444447 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:49:17.447183 (+ 2736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:49:17.450102 (+ 2919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:49:17.450114 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:49:17.450134 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:49:17.450796 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:49:17.451198 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:49:17.451206 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:49:17.451213 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:49:17.451461 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:49:17.451978 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:49:17.451989 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:49:17.452439 (+ 450us) client_negotiation.cc:769] Sending connection context
0810 00:49:17.452733 (+ 294us) client_negotiation.cc:240] Negotiation successful
0810 00:49:17.453325 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":297,"threads_started":1}
I20260810 00:49:17.456459 20923 heartbeater.cc:347] Connected to a master server at 127.28.6.62:44053
I20260810 00:49:17.456780 20923 heartbeater.cc:464] Registering TS with master...
I20260810 00:49:17.457656 20923 heartbeater.cc:511] Master 127.28.6.62:44053 requested a full tablet report, sending...
I20260810 00:49:17.459353 20382 ts_manager.cc:194] Registered new tserver with Master: 92c7bf0b0c8a4ace9fed6b4576ddc4ea (127.28.6.2:34761)
I20260810 00:49:17.460727 20382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.2@KRBTEST.COM'} at 127.28.6.2:41997
W20260810 00:49:17.777997 20927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:49:17.778261 20927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:49:17.778353 20927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:49:17.785277 20927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:49:17.785370 20927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:49:17.785475 20927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.3
I20260810 00:49:17.794562 20927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:39585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.3:46009
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.6.3
--webserver_port=39153
--tserver_master_addrs=127.28.6.62:44053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.3
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:49:17.795725 20927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:49:17.796950 20927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:49:17.809829 20935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:49:17.809865 20937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:49:17.809906 20941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:49:17.810875 20927 server_base.cc:1034] running on GCE node
I20260810 00:49:17.811496 20927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:49:17.812340 20927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:49:17.813530 20927 hybrid_clock.cc:648] HybridClock initialized: now 1786322957813482 us; error 45 us; skew 500 ppm
I20260810 00:49:17.813833 20927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:49:17 dist-test-slave-1nl6 krb5kdc[19398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322957, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:49:17.816926 20927 init.cc:377] Logged in from keytab as kudu/127.28.6.3@KRBTEST.COM (short username kudu)
I20260810 00:49:17.818343 20927 webserver.cc:460] Webserver started at http://127.28.6.3:39153/ using document root <none> and password file <none>
I20260810 00:49:17.819063 20927 fs_manager.cc:359] Metadata directory not provided
I20260810 00:49:17.819193 20927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:49:17.824414 20927 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260810 00:49:17.827128 20950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:49:17.828583 20927 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 00:49:17.828768 20927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/wal
uuid: "c2aa322985374b8092a1f4adb947ab42"
format_stamp: "Formatted at 2026-08-10 00:48:55 on dist-test-slave-1nl6"
server_key: "182ae569985ac8b68141a5764dd00b01"
server_key_iv: "b7fce0030639b16e31f577301e6c7789"
server_key_version: "encryptionkey@0"
I20260810 00:49:17.829810 20927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:49:17.852313 20927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:49:17.856137 20927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:49:17.856415 20927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:49:17.857321 20927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:49:17.858572 20927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:49:17.858690 20927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:49:17.858814 20927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:49:17.858892 20927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:49:17.892603 20927 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.3:46009
I20260810 00:49:17.892691 21140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.3:46009 every 8 connection(s)
I20260810 00:49:17.894099 20927 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
I20260810 00:49:17.894783 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 20927
Aug 10 00:49:17 dist-test-slave-1nl6 krb5kdc[19398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322957, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:49:17.912958 20708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:17.895954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39558 (local address 127.28.6.62:44053)
0810 00:49:17.896213 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0810 00:49:17.896222 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:49:17.897246 (+ 1024us) server_negotiation.cc:407] Connection header received
0810 00:49:17.897501 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:49:17.897509 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:49:17.897610 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:49:17.897735 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:49:17.899934 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:17.900697 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:17.902795 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:17.903519 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:17.903651 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:17.906976 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:49:17.906982 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:49:17.906987 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:49:17.907024 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:49:17.909865 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:49:17.910655 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:49:17.910664 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:49:17.910667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:49:17.910819 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:49:17.911253 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:49:17.911258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:49:17.911262 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:49:17.911612 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:49:17.911840 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0810 00:49:17.912254 (+ 414us) server_negotiation.cc:299] Negotiation successful
0810 00:49:17.912497 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":95}
I20260810 00:49:17.917675 21148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:17.897048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:57371 (local address 127.28.6.62:44053)
0810 00:49:17.899100 (+ 2052us) server_negotiation.cc:206] Beginning negotiation
0810 00:49:17.899104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:49:17.899120 (+ 16us) server_negotiation.cc:407] Connection header received
0810 00:49:17.901131 (+ 2011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:49:17.901135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:49:17.901190 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:49:17.901276 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:49:17.903333 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:17.904384 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:17.906788 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:17.907570 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:17.907716 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:17.911638 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:49:17.911643 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:49:17.911646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:49:17.911707 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:49:17.914972 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:49:17.915742 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:49:17.915748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:49:17.915752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:49:17.915885 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:49:17.916257 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:49:17.916260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:49:17.916262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:49:17.916521 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:49:17.916641 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 00:49:17.917240 (+ 599us) server_negotiation.cc:299] Negotiation successful
0810 00:49:17.917396 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1948,"thread_start_us":542,"threads_started":1}
I20260810 00:49:17.918651 21149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:17.897827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.3:57371)
0810 00:49:17.898713 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:17.898766 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 00:49:17.900835 (+ 2069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:49:17.901554 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:49:17.901566 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:49:17.902288 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:49:17.903138 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:49:17.903167 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:49:17.904548 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:49:17.904554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:49:17.906621 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:49:17.906640 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:49:17.907843 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:49:17.907853 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:49:17.908119 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:17.908717 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:49:17.908757 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:49:17.911414 (+ 2657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:49:17.915100 (+ 3686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:49:17.915107 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:49:17.915118 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:49:17.915618 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:49:17.915979 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:49:17.915982 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:49:17.915985 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:49:17.916155 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:49:17.916627 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:49:17.916632 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:49:17.916918 (+ 286us) client_negotiation.cc:769] Sending connection context
0810 00:49:17.917144 (+ 226us) client_negotiation.cc:240] Negotiation successful
0810 00:49:17.917490 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":336,"threads_started":1}
I20260810 00:49:17.919975 21141 heartbeater.cc:347] Connected to a master server at 127.28.6.62:44053
I20260810 00:49:17.920306 21141 heartbeater.cc:464] Registering TS with master...
I20260810 00:49:17.921039 21141 heartbeater.cc:511] Master 127.28.6.62:44053 requested a full tablet report, sending...
I20260810 00:49:17.922427 20382 ts_manager.cc:194] Registered new tserver with Master: c2aa322985374b8092a1f4adb947ab42 (127.28.6.3:46009)
I20260810 00:49:17.923339 20382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.3@KRBTEST.COM'} at 127.28.6.3:57371
I20260810 00:49:17.927858 28696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 00:49:17.934358 21148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:17.928687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39562 (local address 127.28.6.62:44053)
0810 00:49:17.928838 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 00:49:17.928842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:49:17.929270 (+ 428us) server_negotiation.cc:407] Connection header received
0810 00:49:17.929483 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:49:17.929489 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:49:17.929644 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:49:17.929781 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:49:17.930213 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:17.930980 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:17.932529 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:17.932969 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:17.933099 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:17.933606 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:49:17.933821 (+ 215us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0810 00:49:17.934155 (+ 334us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39562: token expired
Metrics: {"server-negotiator.queue_time_us":45}
W20260810 00:49:17.934494 21148 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39562: token expired
W20260810 00:49:17.935195 20236 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.28.6.62:44053)
I20260810 00:49:17.947832 21148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:17.935668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39568 (local address 127.28.6.62:44053)
0810 00:49:17.935835 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 00:49:17.935841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:49:17.935880 (+ 39us) server_negotiation.cc:407] Connection header received
0810 00:49:17.936068 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:49:17.936074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:49:17.936131 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:49:17.936227 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:49:17.937619 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:17.938294 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:17.939630 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:17.940042 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:17.940180 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:17.942806 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:49:17.942810 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:49:17.942813 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:49:17.942841 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:49:17.945201 (+ 2360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:49:17.945835 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:49:17.945842 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:49:17.945845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:49:17.945970 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:49:17.946355 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:49:17.946360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:49:17.946363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:49:17.946663 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:49:17.946987 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0810 00:49:17.947311 (+ 324us) server_negotiation.cc:299] Negotiation successful
0810 00:49:17.947513 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 00:49:17.950829 20382 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39568:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 00:49:17.958063 20382 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 00:49:17.979701 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:17.969777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37230 (local address 127.28.6.1:35567)
0810 00:49:17.971538 (+ 1761us) server_negotiation.cc:206] Beginning negotiation
0810 00:49:17.971549 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 00:49:17.971580 (+ 31us) server_negotiation.cc:407] Connection header received
0810 00:49:17.971679 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:49:17.971686 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:49:17.971915 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:49:17.972104 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:49:17.972787 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:17.974466 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:17.977603 (+ 3137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:17.978533 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:17.978679 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:17.978947 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0810 00:49:17.979203 (+ 256us) server_negotiation.cc:299] Negotiation successful
0810 00:49:17.979365 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1584,"thread_start_us":1421,"threads_started":1}
I20260810 00:49:17.979830 21160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:17.969472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:35567 (local address 127.0.0.1:37230)
0810 00:49:17.970526 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:17.970584 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 00:49:17.970784 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:49:17.972117 (+ 1333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:49:17.972124 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:49:17.972187 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:49:17.972572 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:49:17.972585 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:49:17.974614 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:49:17.974625 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:49:17.977385 (+ 2760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:49:17.977415 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:49:17.978759 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:49:17.978764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:49:17.978974 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:17.979035 (+ 61us) client_negotiation.cc:769] Sending connection context
0810 00:49:17.979157 (+ 122us) client_negotiation.cc:240] Negotiation successful
0810 00:49:17.979371 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"mutex_wait_us":1,"thread_start_us":547,"threads_started":1}
I20260810 00:49:17.981326 21163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:17.970802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:34761 (local address 127.0.0.1:59280)
0810 00:49:17.972162 (+ 1360us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:17.972190 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 00:49:17.972343 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:49:17.973147 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:49:17.973159 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:49:17.973188 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:49:17.973575 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:49:17.973588 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:49:17.975495 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:49:17.975502 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:49:17.978505 (+ 3003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:49:17.978520 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:49:17.980441 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:49:17.980448 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:49:17.980631 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:17.980675 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 00:49:17.980772 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 00:49:17.980892 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1208,"mutex_wait_us":619,"thread_start_us":1072,"threads_started":1}
I20260810 00:49:17.982049 21164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:17.971173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59280 (local address 127.28.6.2:34761)
0810 00:49:17.971783 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0810 00:49:17.971791 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:49:17.972378 (+ 587us) server_negotiation.cc:407] Connection header received
0810 00:49:17.972485 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:49:17.972493 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:49:17.972727 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:49:17.972908 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:49:17.973754 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:17.975294 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:17.978671 (+ 3377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:17.980197 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:17.980330 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:17.980527 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0810 00:49:17.981540 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0810 00:49:17.981713 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":315,"threads_started":1}
I20260810 00:49:17.984135 21168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:17.973042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:46009 (local address 127.0.0.1:40368)
0810 00:49:17.975078 (+ 2036us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:17.975094 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 00:49:17.975195 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:49:17.976367 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:49:17.976372 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:49:17.976397 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:49:17.976891 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:49:17.976905 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:49:17.979106 (+ 2201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:49:17.979111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:49:17.981674 (+ 2563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:49:17.981687 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:49:17.983397 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:49:17.983403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:49:17.983568 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:17.983618 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 00:49:17.983714 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 00:49:17.983830 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1915,"thread_start_us":1716,"threads_started":1}
I20260810 00:49:17.984303 21165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:17.972609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40368 (local address 127.28.6.3:46009)
0810 00:49:17.975793 (+ 3184us) server_negotiation.cc:206] Beginning negotiation
0810 00:49:17.975802 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:49:17.975845 (+ 43us) server_negotiation.cc:407] Connection header received
0810 00:49:17.975937 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:49:17.975945 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:49:17.976154 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:49:17.977106 (+ 952us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:49:17.977404 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:17.978968 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:17.981818 (+ 2850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:17.983097 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:17.983310 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:17.983542 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0810 00:49:17.983779 (+ 237us) server_negotiation.cc:299] Negotiation successful
0810 00:49:17.983945 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3000,"thread_start_us":2829,"threads_started":1}
I20260810 00:49:17.983955 20602 tablet_service.cc:1467] Processing CreateTablet for tablet 5e10f926d13b464eb954b3c168770d1f (DEFAULT_TABLE table=test-table [id=373fff9680bd402a9c39b37df7be58c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:49:17.985327 20602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e10f926d13b464eb954b3c168770d1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:49:17.985503 20826 tablet_service.cc:1467] Processing CreateTablet for tablet 5e10f926d13b464eb954b3c168770d1f (DEFAULT_TABLE table=test-table [id=373fff9680bd402a9c39b37df7be58c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:49:17.986793 20826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e10f926d13b464eb954b3c168770d1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:49:17.994483 21026 tablet_service.cc:1467] Processing CreateTablet for tablet 5e10f926d13b464eb954b3c168770d1f (DEFAULT_TABLE table=test-table [id=373fff9680bd402a9c39b37df7be58c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:49:17.995873 21026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e10f926d13b464eb954b3c168770d1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:49:17.996197 20706 heartbeater.cc:503] Master 127.28.6.62:44053 was elected leader, sending a full tablet report...
I20260810 00:49:17.999873 21169 tablet_bootstrap.cc:492] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb: Bootstrap starting.
I20260810 00:49:18.000321 21171 tablet_bootstrap.cc:492] T 5e10f926d13b464eb954b3c168770d1f P 92c7bf0b0c8a4ace9fed6b4576ddc4ea: Bootstrap starting.
I20260810 00:49:18.003053 21169 tablet_bootstrap.cc:654] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb: Neither blocks nor log segments found. Creating new log.
I20260810 00:49:18.003774 21171 tablet_bootstrap.cc:654] T 5e10f926d13b464eb954b3c168770d1f P 92c7bf0b0c8a4ace9fed6b4576ddc4ea: Neither blocks nor log segments found. Creating new log.
I20260810 00:49:18.004400 21169 log.cc:824] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb: Log is configured to *not* fsync() on all Append() calls
I20260810 00:49:18.004961 21171 log.cc:824] T 5e10f926d13b464eb954b3c168770d1f P 92c7bf0b0c8a4ace9fed6b4576ddc4ea: Log is configured to *not* fsync() on all Append() calls
I20260810 00:49:18.007529 21169 tablet_bootstrap.cc:492] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb: No bootstrap required, opened a new log
I20260810 00:49:18.007784 21171 tablet_bootstrap.cc:492] T 5e10f926d13b464eb954b3c168770d1f P 92c7bf0b0c8a4ace9fed6b4576ddc4ea: No bootstrap required, opened a new log
I20260810 00:49:18.007793 21169 ts_tablet_manager.cc:1397] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260810 00:49:18.008133 21171 ts_tablet_manager.cc:1397] T 5e10f926d13b464eb954b3c168770d1f P 92c7bf0b0c8a4ace9fed6b4576ddc4ea: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260810 00:49:18.011392 21177 tablet_bootstrap.cc:492] T 5e10f926d13b464eb954b3c168770d1f P c2aa322985374b8092a1f4adb947ab42: Bootstrap starting.
I20260810 00:49:18.015132 21177 tablet_bootstrap.cc:654] T 5e10f926d13b464eb954b3c168770d1f P c2aa322985374b8092a1f4adb947ab42: Neither blocks nor log segments found. Creating new log.
I20260810 00:49:18.015978 21169 raft_consensus.cc:348] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f2b001cfd2415dba06994af259aabb" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 35567 } } peers { permanent_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34761 } } peers { permanent_uuid: "c2aa322985374b8092a1f4adb947ab42" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46009 } }
I20260810 00:49:18.016341 21169 raft_consensus.cc:374] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:49:18.016366 21177 log.cc:824] T 5e10f926d13b464eb954b3c168770d1f P c2aa322985374b8092a1f4adb947ab42: Log is configured to *not* fsync() on all Append() calls
I20260810 00:49:18.016438 21169 raft_consensus.cc:729] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92f2b001cfd2415dba06994af259aabb, State: Initialized, Role: FOLLOWER
I20260810 00:49:18.017112 21169 consensus_queue.cc:260] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f2b001cfd2415dba06994af259aabb" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 35567 } } peers { permanent_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34761 } } peers { permanent_uuid: "c2aa322985374b8092a1f4adb947ab42" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46009 } }
I20260810 00:49:18.018036 21171 raft_consensus.cc:348] T 5e10f926d13b464eb954b3c168770d1f P 92c7bf0b0c8a4ace9fed6b4576ddc4ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f2b001cfd2415dba06994af259aabb" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 35567 } } peers { permanent_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34761 } } peers { permanent_uuid: "c2aa322985374b8092a1f4adb947ab42" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46009 } }
I20260810 00:49:18.018472 21171 raft_consensus.cc:374] T 5e10f926d13b464eb954b3c168770d1f P 92c7bf0b0c8a4ace9fed6b4576ddc4ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:49:18.018570 21171 raft_consensus.cc:729] T 5e10f926d13b464eb954b3c168770d1f P 92c7bf0b0c8a4ace9fed6b4576ddc4ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92c7bf0b0c8a4ace9fed6b4576ddc4ea, State: Initialized, Role: FOLLOWER
I20260810 00:49:18.019435 21171 consensus_queue.cc:260] T 5e10f926d13b464eb954b3c168770d1f P 92c7bf0b0c8a4ace9fed6b4576ddc4ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f2b001cfd2415dba06994af259aabb" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 35567 } } peers { permanent_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34761 } } peers { permanent_uuid: "c2aa322985374b8092a1f4adb947ab42" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46009 } }
I20260810 00:49:18.019759 21177 tablet_bootstrap.cc:492] T 5e10f926d13b464eb954b3c168770d1f P c2aa322985374b8092a1f4adb947ab42: No bootstrap required, opened a new log
I20260810 00:49:18.020025 21177 ts_tablet_manager.cc:1397] T 5e10f926d13b464eb954b3c168770d1f P c2aa322985374b8092a1f4adb947ab42: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260810 00:49:18.020399 21169 ts_tablet_manager.cc:1428] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
I20260810 00:49:18.024188 20923 heartbeater.cc:503] Master 127.28.6.62:44053 was elected leader, sending a full tablet report...
I20260810 00:49:18.024190 21171 ts_tablet_manager.cc:1428] T 5e10f926d13b464eb954b3c168770d1f P 92c7bf0b0c8a4ace9fed6b4576ddc4ea: Time spent starting tablet: real 0.016s user 0.012s sys 0.002s
I20260810 00:49:18.029085 21177 raft_consensus.cc:348] T 5e10f926d13b464eb954b3c168770d1f P c2aa322985374b8092a1f4adb947ab42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f2b001cfd2415dba06994af259aabb" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 35567 } } peers { permanent_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34761 } } peers { permanent_uuid: "c2aa322985374b8092a1f4adb947ab42" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46009 } }
I20260810 00:49:18.029409 21177 raft_consensus.cc:374] T 5e10f926d13b464eb954b3c168770d1f P c2aa322985374b8092a1f4adb947ab42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:49:18.029523 21177 raft_consensus.cc:729] T 5e10f926d13b464eb954b3c168770d1f P c2aa322985374b8092a1f4adb947ab42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2aa322985374b8092a1f4adb947ab42, State: Initialized, Role: FOLLOWER
I20260810 00:49:18.030023 21177 consensus_queue.cc:260] T 5e10f926d13b464eb954b3c168770d1f P c2aa322985374b8092a1f4adb947ab42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f2b001cfd2415dba06994af259aabb" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 35567 } } peers { permanent_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34761 } } peers { permanent_uuid: "c2aa322985374b8092a1f4adb947ab42" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46009 } }
I20260810 00:49:18.030995 21141 heartbeater.cc:503] Master 127.28.6.62:44053 was elected leader, sending a full tablet report...
I20260810 00:49:18.031538 21177 ts_tablet_manager.cc:1428] T 5e10f926d13b464eb954b3c168770d1f P c2aa322985374b8092a1f4adb947ab42: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
W20260810 00:49:18.147046 21143 tablet.cc:2369] T 5e10f926d13b464eb954b3c168770d1f P c2aa322985374b8092a1f4adb947ab42: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 00:49:18.181659 20925 tablet.cc:2369] T 5e10f926d13b464eb954b3c168770d1f P 92c7bf0b0c8a4ace9fed6b4576ddc4ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:49:18.193646 21179 raft_consensus.cc:482] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:49:18.193966 21179 raft_consensus.cc:504] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f2b001cfd2415dba06994af259aabb" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 35567 } } peers { permanent_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34761 } } peers { permanent_uuid: "c2aa322985374b8092a1f4adb947ab42" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46009 } }
I20260810 00:49:18.195955 21179 leader_election.cc:288] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92c7bf0b0c8a4ace9fed6b4576ddc4ea (127.28.6.2:34761), c2aa322985374b8092a1f4adb947ab42 (127.28.6.3:46009)
I20260810 00:49:18.195972 21180 raft_consensus.cc:482] T 5e10f926d13b464eb954b3c168770d1f P 92c7bf0b0c8a4ace9fed6b4576ddc4ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:49:18.196313 21180 raft_consensus.cc:504] T 5e10f926d13b464eb954b3c168770d1f P 92c7bf0b0c8a4ace9fed6b4576ddc4ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f2b001cfd2415dba06994af259aabb" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 35567 } } peers { permanent_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34761 } } peers { permanent_uuid: "c2aa322985374b8092a1f4adb947ab42" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46009 } }
I20260810 00:49:18.198508 21180 leader_election.cc:288] T 5e10f926d13b464eb954b3c168770d1f P 92c7bf0b0c8a4ace9fed6b4576ddc4ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92f2b001cfd2415dba06994af259aabb (127.28.6.1:35567), c2aa322985374b8092a1f4adb947ab42 (127.28.6.3:46009)
I20260810 00:49:18.203200 21181 raft_consensus.cc:482] T 5e10f926d13b464eb954b3c168770d1f P c2aa322985374b8092a1f4adb947ab42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:49:18.203528 21181 raft_consensus.cc:504] T 5e10f926d13b464eb954b3c168770d1f P c2aa322985374b8092a1f4adb947ab42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f2b001cfd2415dba06994af259aabb" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 35567 } } peers { permanent_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34761 } } peers { permanent_uuid: "c2aa322985374b8092a1f4adb947ab42" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46009 } }
I20260810 00:49:18.205554 21181 leader_election.cc:288] T 5e10f926d13b464eb954b3c168770d1f P c2aa322985374b8092a1f4adb947ab42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92f2b001cfd2415dba06994af259aabb (127.28.6.1:35567), 92c7bf0b0c8a4ace9fed6b4576ddc4ea (127.28.6.2:34761)
I20260810 00:49:18.207484 21165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:18.197386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:52601 (local address 127.28.6.3:46009)
0810 00:49:18.197611 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 00:49:18.197618 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:49:18.198383 (+ 765us) server_negotiation.cc:407] Connection header received
0810 00:49:18.198635 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:49:18.198641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:49:18.198731 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:49:18.198793 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:49:18.199694 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:18.200824 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:18.204968 (+ 4144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:18.206285 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:18.206379 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:18.206465 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 00:49:18.207117 (+ 652us) server_negotiation.cc:299] Negotiation successful
0810 00:49:18.207246 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 00:49:18.208437 21183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:18.197237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:46009 (local address 127.28.6.1:52601)
0810 00:49:18.198301 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:18.198319 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 00:49:18.198486 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:49:18.199085 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:49:18.199089 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:49:18.199114 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:49:18.199478 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:49:18.199489 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:49:18.201007 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:49:18.201012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:49:18.204792 (+ 3780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:49:18.204808 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:49:18.206729 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:49:18.206738 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:49:18.206924 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:18.206958 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 00:49:18.207874 (+ 916us) client_negotiation.cc:240] Negotiation successful
0810 00:49:18.207997 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":936,"mutex_wait_us":81,"thread_start_us":712,"threads_started":1}
I20260810 00:49:18.209228 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:18.198863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:51899 (local address 127.28.6.1:35567)
0810 00:49:18.199134 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0810 00:49:18.199142 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:49:18.199763 (+ 621us) server_negotiation.cc:407] Connection header received
0810 00:49:18.200059 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:49:18.200064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:49:18.200135 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:49:18.200199 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:49:18.201187 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:18.202733 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:18.206838 (+ 4105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:18.207993 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:18.208119 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:18.208238 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 00:49:18.208854 (+ 616us) server_negotiation.cc:299] Negotiation successful
0810 00:49:18.208992 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":105}
I20260810 00:49:18.210063 21164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:18.196327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:33941 (local address 127.28.6.2:34761)
0810 00:49:18.196524 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 00:49:18.196529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:49:18.197200 (+ 671us) server_negotiation.cc:407] Connection header received
0810 00:49:18.197481 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:49:18.197492 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:49:18.197568 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:49:18.197655 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:49:18.198379 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:18.199592 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:18.202292 (+ 2700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:18.203501 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:18.203639 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:18.203733 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 00:49:18.209685 (+ 5952us) server_negotiation.cc:299] Negotiation successful
0810 00:49:18.209810 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260810 00:49:18.210753 20707 tablet.cc:2369] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:49:18.212633 21182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:18.196405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:34761 (local address 127.28.6.1:33941)
0810 00:49:18.197075 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:18.197090 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 00:49:18.197334 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:49:18.197704 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:49:18.197709 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:49:18.197737 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:49:18.198181 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:49:18.198195 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:49:18.199779 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:49:18.199785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:49:18.202075 (+ 2290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:49:18.202092 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:49:18.203926 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:49:18.203934 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:49:18.209528 (+ 5594us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:18.209564 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 00:49:18.209660 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 00:49:18.209793 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":5217,"thread_start_us":398,"threads_started":1}
I20260810 00:49:18.216786 20853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e10f926d13b464eb954b3c168770d1f" candidate_uuid: "92f2b001cfd2415dba06994af259aabb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea" is_pre_election: true
I20260810 00:49:18.217288 20853 raft_consensus.cc:2454] T 5e10f926d13b464eb954b3c168770d1f P 92c7bf0b0c8a4ace9fed6b4576ddc4ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92f2b001cfd2415dba06994af259aabb in term 0.
I20260810 00:49:18.217450 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:18.207277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:59671 (local address 127.28.6.1:35567)
0810 00:49:18.209488 (+ 2211us) server_negotiation.cc:206] Beginning negotiation
0810 00:49:18.209493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:49:18.209526 (+ 33us) server_negotiation.cc:407] Connection header received
0810 00:49:18.209578 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:49:18.209582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:49:18.210377 (+ 795us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:49:18.210957 (+ 580us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:49:18.211093 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:18.212144 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:18.214696 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:18.215934 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:18.216922 (+ 988us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:18.217017 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 00:49:18.217063 (+ 46us) server_negotiation.cc:299] Negotiation successful
0810 00:49:18.217172 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2123,"threads_started":1}
I20260810 00:49:18.217962 21149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:18.206031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:35567 (local address 127.28.6.3:59671)
0810 00:49:18.206834 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:18.206854 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 00:49:18.206978 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:49:18.210562 (+ 3584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:49:18.210566 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:49:18.210593 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:49:18.210797 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:49:18.210809 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:49:18.212348 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:49:18.212354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:49:18.214486 (+ 2132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:49:18.214504 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:49:18.216450 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:49:18.216456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:49:18.216667 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:18.216713 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 00:49:18.216776 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 00:49:18.217675 (+ 899us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"mutex_wait_us":717}
I20260810 00:49:18.217994 20519 leader_election.cc:302] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92c7bf0b0c8a4ace9fed6b4576ddc4ea, 92f2b001cfd2415dba06994af259aabb; no voters:
I20260810 00:49:18.218463 21179 raft_consensus.cc:2781] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 00:49:18.218631 21179 raft_consensus.cc:482] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 00:49:18.218757 21179 raft_consensus.cc:3037] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:49:18.219857 21165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:18.200107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:38667 (local address 127.28.6.3:46009)
0810 00:49:18.207668 (+ 7561us) server_negotiation.cc:206] Beginning negotiation
0810 00:49:18.207674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:49:18.209248 (+ 1574us) server_negotiation.cc:407] Connection header received
0810 00:49:18.211729 (+ 2481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:49:18.211734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:49:18.211813 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:49:18.211870 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:49:18.213029 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:18.214225 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:18.216622 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:18.218156 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:18.218291 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:18.218419 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 00:49:18.219424 (+ 1005us) server_negotiation.cc:299] Negotiation successful
0810 00:49:18.219577 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":263,"server-negotiator.queue_time_us":7475,"threads_started":1}
I20260810 00:49:18.220605 21185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:18.199754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:46009 (local address 127.28.6.2:38667)
0810 00:49:18.209152 (+ 9398us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:18.209171 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 00:49:18.210338 (+ 1167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:49:18.212028 (+ 1690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:49:18.212032 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:49:18.212055 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:49:18.212458 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:49:18.212472 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:49:18.214402 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:49:18.214408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:49:18.216434 (+ 2026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:49:18.216448 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:49:18.219043 (+ 2595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:49:18.219051 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:49:18.219237 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:18.219270 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 00:49:18.220145 (+ 875us) client_negotiation.cc:240] Negotiation successful
0810 00:49:18.220275 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9288,"thread_start_us":9090,"threads_started":1}
I20260810 00:49:18.221385 21184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:18.198634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:35567 (local address 127.28.6.2:51899)
0810 00:49:18.199671 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:18.199692 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 00:49:18.199885 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:49:18.200461 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:49:18.200468 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:49:18.200501 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:49:18.200999 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:49:18.201014 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:49:18.204569 (+ 3555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:49:18.204574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:49:18.206669 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:49:18.206687 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:49:18.208460 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:49:18.208467 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:49:18.208644 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:18.208682 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 00:49:18.218665 (+ 9983us) client_negotiation.cc:240] Negotiation successful
0810 00:49:18.221020 (+ 2355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":855,"mutex_wait_us":1969,"thread_start_us":677,"threads_started":1}
I20260810 00:49:18.222632 20636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e10f926d13b464eb954b3c168770d1f" candidate_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92f2b001cfd2415dba06994af259aabb" is_pre_election: true
I20260810 00:49:18.222798 21058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e10f926d13b464eb954b3c168770d1f" candidate_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2aa322985374b8092a1f4adb947ab42" is_pre_election: true
I20260810 00:49:18.223371 21058 raft_consensus.cc:2454] T 5e10f926d13b464eb954b3c168770d1f P c2aa322985374b8092a1f4adb947ab42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92c7bf0b0c8a4ace9fed6b4576ddc4ea in term 0.
I20260810 00:49:18.224376 21179 raft_consensus.cc:504] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f2b001cfd2415dba06994af259aabb" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 35567 } } peers { permanent_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34761 } } peers { permanent_uuid: "c2aa322985374b8092a1f4adb947ab42" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46009 } }
I20260810 00:49:18.224500 20748 leader_election.cc:302] T 5e10f926d13b464eb954b3c168770d1f P 92c7bf0b0c8a4ace9fed6b4576ddc4ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92c7bf0b0c8a4ace9fed6b4576ddc4ea, c2aa322985374b8092a1f4adb947ab42; no voters:
I20260810 00:49:18.224896 20636 raft_consensus.cc:2379] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 92c7bf0b0c8a4ace9fed6b4576ddc4ea in current term 1: Already voted for candidate 92f2b001cfd2415dba06994af259aabb in this term.
I20260810 00:49:18.224918 21180 raft_consensus.cc:2781] T 5e10f926d13b464eb954b3c168770d1f P 92c7bf0b0c8a4ace9fed6b4576ddc4ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 00:49:18.224756 20635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e10f926d13b464eb954b3c168770d1f" candidate_uuid: "c2aa322985374b8092a1f4adb947ab42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92f2b001cfd2415dba06994af259aabb" is_pre_election: true
I20260810 00:49:18.225059 21180 raft_consensus.cc:482] T 5e10f926d13b464eb954b3c168770d1f P 92c7bf0b0c8a4ace9fed6b4576ddc4ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 00:49:18.225150 21180 raft_consensus.cc:3037] T 5e10f926d13b464eb954b3c168770d1f P 92c7bf0b0c8a4ace9fed6b4576ddc4ea [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:49:18.225680 21058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e10f926d13b464eb954b3c168770d1f" candidate_uuid: "92f2b001cfd2415dba06994af259aabb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2aa322985374b8092a1f4adb947ab42" is_pre_election: true
I20260810 00:49:18.225930 21058 raft_consensus.cc:2454] T 5e10f926d13b464eb954b3c168770d1f P c2aa322985374b8092a1f4adb947ab42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92f2b001cfd2415dba06994af259aabb in term 0.
I20260810 00:49:18.227172 21164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:18.207662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:54413 (local address 127.28.6.2:34761)
0810 00:49:18.210299 (+ 2637us) server_negotiation.cc:206] Beginning negotiation
0810 00:49:18.210304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:49:18.218330 (+ 8026us) server_negotiation.cc:407] Connection header received
0810 00:49:18.218574 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:49:18.218579 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:49:18.218646 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:49:18.218703 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:49:18.219487 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:18.220564 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:18.224183 (+ 3619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:18.225868 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:18.226004 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:18.226096 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 00:49:18.226717 (+ 621us) server_negotiation.cc:299] Negotiation successful
0810 00:49:18.226849 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":187,"server-negotiator.queue_time_us":2546,"thread_start_us":5193,"threads_started":1}
I20260810 00:49:18.227381 21179 leader_election.cc:288] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [CANDIDATE]: Term 1 election: Requested vote from peers 92c7bf0b0c8a4ace9fed6b4576ddc4ea (127.28.6.2:34761), c2aa322985374b8092a1f4adb947ab42 (127.28.6.3:46009)
I20260810 00:49:18.228061 21149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:18.206417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:34761 (local address 127.28.6.3:54413)
0810 00:49:18.218217 (+ 11800us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:18.218236 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 00:49:18.218421 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:49:18.218846 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:49:18.218852 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:49:18.218878 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:49:18.219279 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:49:18.219293 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:49:18.220780 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:49:18.220785 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:49:18.223972 (+ 3187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:49:18.223995 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:49:18.226276 (+ 2281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:49:18.226282 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:49:18.226486 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:18.226519 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 00:49:18.227619 (+ 1100us) client_negotiation.cc:240] Negotiation successful
0810 00:49:18.227764 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11689,"thread_start_us":17457,"threads_started":1}
I20260810 00:49:18.228056 20853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e10f926d13b464eb954b3c168770d1f" candidate_uuid: "92f2b001cfd2415dba06994af259aabb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea"
I20260810 00:49:18.228107 21058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e10f926d13b464eb954b3c168770d1f" candidate_uuid: "92f2b001cfd2415dba06994af259aabb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2aa322985374b8092a1f4adb947ab42"
I20260810 00:49:18.228310 21058 raft_consensus.cc:3037] T 5e10f926d13b464eb954b3c168770d1f P c2aa322985374b8092a1f4adb947ab42 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:49:18.228525 21180 raft_consensus.cc:504] T 5e10f926d13b464eb954b3c168770d1f P 92c7bf0b0c8a4ace9fed6b4576ddc4ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f2b001cfd2415dba06994af259aabb" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 35567 } } peers { permanent_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34761 } } peers { permanent_uuid: "c2aa322985374b8092a1f4adb947ab42" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46009 } }
I20260810 00:49:18.229082 20852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e10f926d13b464eb954b3c168770d1f" candidate_uuid: "c2aa322985374b8092a1f4adb947ab42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea" is_pre_election: true
I20260810 00:49:18.229350 21180 leader_election.cc:288] T 5e10f926d13b464eb954b3c168770d1f P 92c7bf0b0c8a4ace9fed6b4576ddc4ea [CANDIDATE]: Term 1 election: Requested vote from peers 92f2b001cfd2415dba06994af259aabb (127.28.6.1:35567), c2aa322985374b8092a1f4adb947ab42 (127.28.6.3:46009)
I20260810 00:49:18.229694 20636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e10f926d13b464eb954b3c168770d1f" candidate_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92f2b001cfd2415dba06994af259aabb"
I20260810 00:49:18.229830 20853 raft_consensus.cc:2379] T 5e10f926d13b464eb954b3c168770d1f P 92c7bf0b0c8a4ace9fed6b4576ddc4ea [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 92f2b001cfd2415dba06994af259aabb in current term 1: Already voted for candidate 92c7bf0b0c8a4ace9fed6b4576ddc4ea in this term.
I20260810 00:49:18.229944 20636 raft_consensus.cc:2379] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 92c7bf0b0c8a4ace9fed6b4576ddc4ea in current term 1: Already voted for candidate 92f2b001cfd2415dba06994af259aabb in this term.
I20260810 00:49:18.230192 21057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e10f926d13b464eb954b3c168770d1f" candidate_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2aa322985374b8092a1f4adb947ab42"
I20260810 00:49:18.230573 20956 leader_election.cc:302] T 5e10f926d13b464eb954b3c168770d1f P c2aa322985374b8092a1f4adb947ab42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2aa322985374b8092a1f4adb947ab42; no voters: 92c7bf0b0c8a4ace9fed6b4576ddc4ea, 92f2b001cfd2415dba06994af259aabb
I20260810 00:49:18.231331 20748 leader_election.cc:302] T 5e10f926d13b464eb954b3c168770d1f P 92c7bf0b0c8a4ace9fed6b4576ddc4ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92c7bf0b0c8a4ace9fed6b4576ddc4ea; no voters: 92f2b001cfd2415dba06994af259aabb, c2aa322985374b8092a1f4adb947ab42
I20260810 00:49:18.231593 21180 raft_consensus.cc:2726] T 5e10f926d13b464eb954b3c168770d1f P 92c7bf0b0c8a4ace9fed6b4576ddc4ea [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 00:49:18.231956 21058 raft_consensus.cc:2454] T 5e10f926d13b464eb954b3c168770d1f P c2aa322985374b8092a1f4adb947ab42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92f2b001cfd2415dba06994af259aabb in term 1.
I20260810 00:49:18.232179 21181 raft_consensus.cc:2726] T 5e10f926d13b464eb954b3c168770d1f P c2aa322985374b8092a1f4adb947ab42 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 00:49:18.232372 20519 leader_election.cc:302] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92f2b001cfd2415dba06994af259aabb, c2aa322985374b8092a1f4adb947ab42; no voters: 92c7bf0b0c8a4ace9fed6b4576ddc4ea
I20260810 00:49:18.232659 21179 raft_consensus.cc:2781] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:49:18.233168 21179 raft_consensus.cc:686] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [term 1 LEADER]: Becoming Leader. State: Replica: 92f2b001cfd2415dba06994af259aabb, State: Running, Role: LEADER
I20260810 00:49:18.233697 21179 consensus_queue.cc:237] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f2b001cfd2415dba06994af259aabb" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 35567 } } peers { permanent_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34761 } } peers { permanent_uuid: "c2aa322985374b8092a1f4adb947ab42" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46009 } }
I20260810 00:49:18.237884 20380 catalog_manager.cc:5495] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb reported cstate change: term changed from 0 to 1, leader changed from <none> to 92f2b001cfd2415dba06994af259aabb (127.28.6.1). New cstate: current_term: 1 leader_uuid: "92f2b001cfd2415dba06994af259aabb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f2b001cfd2415dba06994af259aabb" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 35567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2aa322985374b8092a1f4adb947ab42" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46009 } health_report { overall_health: UNKNOWN } } }
I20260810 00:49:18.271281 21188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:18.266127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37234 (local address 127.28.6.1:35567)
0810 00:49:18.266321 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 00:49:18.266325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:49:18.266353 (+ 28us) server_negotiation.cc:407] Connection header received
0810 00:49:18.266438 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:49:18.266442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:49:18.266509 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:49:18.266637 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:49:18.267032 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:18.267785 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:18.269138 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:18.269609 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:18.269724 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:18.270200 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:49:18.270725 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:49:18.270829 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 00:49:18.270966 (+ 137us) server_negotiation.cc:299] Negotiation successful
0810 00:49:18.271026 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 00:49:18.279601 21058 raft_consensus.cc:1261] T 5e10f926d13b464eb954b3c168770d1f P c2aa322985374b8092a1f4adb947ab42 [term 1 FOLLOWER]: Refusing update from remote peer 92f2b001cfd2415dba06994af259aabb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:49:18.279630 20853 raft_consensus.cc:1261] T 5e10f926d13b464eb954b3c168770d1f P 92c7bf0b0c8a4ace9fed6b4576ddc4ea [term 1 FOLLOWER]: Refusing update from remote peer 92f2b001cfd2415dba06994af259aabb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:49:18.280405 21193 consensus_queue.cc:1035] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2aa322985374b8092a1f4adb947ab42" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:49:18.280710 21179 consensus_queue.cc:1035] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [LEADER]: Connected to new peer: Peer: permanent_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:49:18.296792 21196 mvcc.cc:205] Tried to move back new op lower bound from 7316778837102882816 to 7316778836928843776. Current Snapshot: MvccSnapshot[applied={T|T < 7316778837102882816}]
I20260810 00:49:18.306193 20382 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:49:18.306473 20382 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:49:18.309736 20382 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb: Sending DeleteTablet for 3 replicas of tablet 5e10f926d13b464eb954b3c168770d1f
I20260810 00:49:18.310784 20602 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e10f926d13b464eb954b3c168770d1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:49:18 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:37230
I20260810 00:49:18.311077 21026 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e10f926d13b464eb954b3c168770d1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:49:18 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:40368
I20260810 00:49:18.311736 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 20447
I20260810 00:49:18.311776 20826 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e10f926d13b464eb954b3c168770d1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:49:18 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:59280
I20260810 00:49:18.319659 21214 tablet_replica.cc:330] T 5e10f926d13b464eb954b3c168770d1f P c2aa322985374b8092a1f4adb947ab42: stopping tablet replica
I20260810 00:49:18.320350 21215 tablet_replica.cc:330] T 5e10f926d13b464eb954b3c168770d1f P 92c7bf0b0c8a4ace9fed6b4576ddc4ea: stopping tablet replica
I20260810 00:49:18.320801 21215 raft_consensus.cc:2229] T 5e10f926d13b464eb954b3c168770d1f P 92c7bf0b0c8a4ace9fed6b4576ddc4ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:49:18.321349 21215 raft_consensus.cc:2258] T 5e10f926d13b464eb954b3c168770d1f P 92c7bf0b0c8a4ace9fed6b4576ddc4ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:49:18.322026 21214 raft_consensus.cc:2229] T 5e10f926d13b464eb954b3c168770d1f P c2aa322985374b8092a1f4adb947ab42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:49:18.322510 21214 raft_consensus.cc:2258] T 5e10f926d13b464eb954b3c168770d1f P c2aa322985374b8092a1f4adb947ab42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:49:18.325066 21215 ts_tablet_manager.cc:1905] T 5e10f926d13b464eb954b3c168770d1f P 92c7bf0b0c8a4ace9fed6b4576ddc4ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:49:18.326925 21214 ts_tablet_manager.cc:1905] T 5e10f926d13b464eb954b3c168770d1f P c2aa322985374b8092a1f4adb947ab42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:49:18.334172 21215 ts_tablet_manager.cc:1918] T 5e10f926d13b464eb954b3c168770d1f P 92c7bf0b0c8a4ace9fed6b4576ddc4ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:49:18.334398 21215 log.cc:1196] T 5e10f926d13b464eb954b3c168770d1f P 92c7bf0b0c8a4ace9fed6b4576ddc4ea: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/wal/wals/5e10f926d13b464eb954b3c168770d1f
I20260810 00:49:18.335026 21215 ts_tablet_manager.cc:1939] T 5e10f926d13b464eb954b3c168770d1f P 92c7bf0b0c8a4ace9fed6b4576ddc4ea: Deleting consensus metadata
I20260810 00:49:18.336200 21188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:18.312585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37236 (local address 127.28.6.1:35567)
0810 00:49:18.312880 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0810 00:49:18.312894 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 00:49:18.312922 (+ 28us) server_negotiation.cc:407] Connection header received
0810 00:49:18.312993 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:49:18.313001 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:49:18.313096 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:49:18.313241 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:49:18.315201 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:18.316699 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:18.318539 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:18.319511 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:18.319657 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:18.324429 (+ 4772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:49:18.324442 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:49:18.324458 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:49:18.324496 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:49:18.328993 (+ 4497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:49:18.333289 (+ 4296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:49:18.333298 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:49:18.333302 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:49:18.333478 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:49:18.333973 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:49:18.333979 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:49:18.333983 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:49:18.334800 (+ 817us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:49:18.335167 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0810 00:49:18.335488 (+ 321us) server_negotiation.cc:299] Negotiation successful
0810 00:49:18.335734 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
I20260810 00:49:18.337086 20364 catalog_manager.cc:4841] TS 92c7bf0b0c8a4ace9fed6b4576ddc4ea (127.28.6.2:34761): tablet 5e10f926d13b464eb954b3c168770d1f (table test-table [id=373fff9680bd402a9c39b37df7be58c4]) successfully deleted
I20260810 00:49:18.337803 20698 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:49:18.339222 21214 ts_tablet_manager.cc:1918] T 5e10f926d13b464eb954b3c168770d1f P c2aa322985374b8092a1f4adb947ab42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:49:18.339390 21214 log.cc:1196] T 5e10f926d13b464eb954b3c168770d1f P c2aa322985374b8092a1f4adb947ab42: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/wal/wals/5e10f926d13b464eb954b3c168770d1f
I20260810 00:49:18.340052 21214 ts_tablet_manager.cc:1939] T 5e10f926d13b464eb954b3c168770d1f P c2aa322985374b8092a1f4adb947ab42: Deleting consensus metadata
I20260810 00:49:18.341979 20364 catalog_manager.cc:4841] TS c2aa322985374b8092a1f4adb947ab42 (127.28.6.3:46009): tablet 5e10f926d13b464eb954b3c168770d1f (table test-table [id=373fff9680bd402a9c39b37df7be58c4]) successfully deleted
I20260810 00:49:18.441840 21213 tablet_replica.cc:330] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb: stopping tablet replica
I20260810 00:49:18.442299 21213 raft_consensus.cc:2229] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [term 1 LEADER]: Raft consensus shutting down.
I20260810 00:49:18.443029 21213 raft_consensus.cc:2258] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:49:18.444923 21213 ts_tablet_manager.cc:1905] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:49:18.447103 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 20447
W20260810 00:49:18.484705 20364 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.6.1:35567 (error 108)
I20260810 00:49:18.485668 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 20710
I20260810 00:49:18.487176 21221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:18.486091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:35567 (local address 127.0.0.1:37250)
0810 00:49:18.486824 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:18.487000 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.1:35567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":539,"threads_started":1}
W20260810 00:49:18.487484 20364 catalog_manager.cc:4584] TS 92f2b001cfd2415dba06994af259aabb (127.28.6.1:35567): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5e10f926d13b464eb954b3c168770d1f: Network error: Client connection negotiation failed: client connection to 127.28.6.1:35567: connect: Connection refused (error 111)
I20260810 00:49:18.499655 21164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:18.486441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59296 (local address 127.28.6.2:34761)
0810 00:49:18.486622 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 00:49:18.486629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:49:18.486816 (+ 187us) server_negotiation.cc:407] Connection header received
0810 00:49:18.486978 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:49:18.486983 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:49:18.487085 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:49:18.487197 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:49:18.488703 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:18.489768 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:18.491261 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:18.491882 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:18.492025 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:18.494339 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:49:18.494351 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:49:18.494365 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:49:18.494403 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:49:18.497203 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:49:18.497773 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:49:18.497780 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:49:18.497782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:49:18.497911 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:49:18.498259 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:49:18.498264 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:49:18.498266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:49:18.498600 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:49:18.498832 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0810 00:49:18.499100 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 00:49:18.499274 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 00:49:18.500360 20917 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:49:18.541020 21221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:18.540558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:35567 (local address 127.0.0.1:37266)
0810 00:49:18.540762 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:18.540862 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.1:35567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260810 00:49:18.542112 21221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:18.541716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:35567 (local address 127.0.0.1:37276)
0810 00:49:18.541875 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:18.541956 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.1:35567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 00:49:18.593472 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 20710
I20260810 00:49:18.602530 21221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:18.601951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:35567 (local address 127.0.0.1:37278)
0810 00:49:18.602205 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:18.602336 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.1:35567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260810 00:49:18.603668 21221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:18.603217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:35567 (local address 127.0.0.1:37282)
0810 00:49:18.603390 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:18.603483 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.1:35567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 00:49:18.621120 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 20927
I20260810 00:49:18.635426 21186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:18.621902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40370 (local address 127.28.6.3:46009)
0810 00:49:18.622096 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 00:49:18.622103 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:49:18.622133 (+ 30us) server_negotiation.cc:407] Connection header received
0810 00:49:18.622378 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:49:18.622385 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:49:18.622483 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:49:18.622608 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:49:18.624106 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:18.624823 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:18.626107 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:18.626528 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:18.626671 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:18.629552 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:49:18.629567 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:49:18.629584 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:49:18.629616 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:49:18.632382 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:49:18.633145 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:49:18.633153 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:49:18.633156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:49:18.633345 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:49:18.633784 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:49:18.633790 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:49:18.633793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:49:18.634153 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:49:18.634468 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0810 00:49:18.634788 (+ 320us) server_negotiation.cc:299] Negotiation successful
0810 00:49:18.635069 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 00:49:18.636102 21129 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:49:18.684193 21221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:18.683734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:35567 (local address 127.0.0.1:37290)
0810 00:49:18.683959 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:18.684046 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.1:35567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260810 00:49:18.685282 21221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:18.684863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:35567 (local address 127.0.0.1:37292)
0810 00:49:18.685065 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:18.685142 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.1:35567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260810 00:49:18.725629 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 20927
I20260810 00:49:18.751467 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 20315
I20260810 00:49:18.752696 20432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:49:18.863766 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 20315
I20260810 00:49:38.897585 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:44053
--webserver_interface=127.28.6.62
--webserver_port=36879
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:39585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:44053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:49:39.238938 21229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:49:39.239269 21229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:49:39.239425 21229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:49:39.246805 21229 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 00:49:39.246963 21229 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 00:49:39.247071 21229 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 00:49:39.247176 21229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:49:39.247291 21229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:49:39.247409 21229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:49:39.247498 21229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:49:39.247579 21229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:49:39.257324 21229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:39585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:44053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:44053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=36879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:49:39.258545 21229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:49:39.259908 21229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:49:39.272243 21240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:49:39.272249 21237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:49:39.272408 21242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:49:39.272823 21229 server_base.cc:1034] running on GCE node
I20260810 00:49:39.273658 21229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:49:39.275005 21229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:49:39.276222 21229 hybrid_clock.cc:648] HybridClock initialized: now 1786322979276156 us; error 42 us; skew 500 ppm
I20260810 00:49:39.276561 21229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:49:39 dist-test-slave-1nl6 krb5kdc[19398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322979, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:49:39.283797 21229 init.cc:377] Logged in from keytab as kudu/127.28.6.62@KRBTEST.COM (short username kudu)
I20260810 00:49:39.285303 21229 webserver.cc:460] Webserver started at http://127.28.6.62:36879/ using document root <none> and password file <none>
I20260810 00:49:39.285961 21229 fs_manager.cc:359] Metadata directory not provided
I20260810 00:49:39.286049 21229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:49:39.291252 21229 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 00:49:39.293977 21257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:49:39.295310 21229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 00:49:39.295450 21229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "491fe8c474e84bd9b275042408d6adbb"
format_stamp: "Formatted at 2026-08-10 00:48:53 on dist-test-slave-1nl6"
server_key: "9782b27bfcedfd6c8521961244e154bf"
server_key_iv: "677b1b6c806eb2c5e6812568cbcc18f1"
server_key_version: "encryptionkey@0"
I20260810 00:49:39.296350 21229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:49:39.319669 21229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:49:39.323555 21229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:49:39.323787 21229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:49:39.342461 21229 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:44053
I20260810 00:49:39.342547 21346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:44053 every 8 connection(s)
I20260810 00:49:39.343930 21229 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:49:39.347788 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 21229
I20260810 00:49:39.348659 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.1:35567
--local_ip_for_outbound_sockets=127.28.6.1
--tserver_master_addrs=127.28.6.62:44053
--webserver_port=46097
--webserver_interface=127.28.6.1
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:39585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 00:49:39.351349 21347 sys_catalog.cc:263] Verifying existing consensus state
I20260810 00:49:39.354597 21347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb: Bootstrap starting.
I20260810 00:49:39.375056 21347 log.cc:824] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb: Log is configured to *not* fsync() on all Append() calls
I20260810 00:49:39.392848 21347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 00:49:39.393502 21347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb: Bootstrap complete.
I20260810 00:49:39.403244 21347 raft_consensus.cc:348] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491fe8c474e84bd9b275042408d6adbb" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 44053 } }
I20260810 00:49:39.404157 21347 raft_consensus.cc:729] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491fe8c474e84bd9b275042408d6adbb, State: Initialized, Role: FOLLOWER
I20260810 00:49:39.404819 21347 consensus_queue.cc:260] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491fe8c474e84bd9b275042408d6adbb" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 44053 } }
I20260810 00:49:39.405076 21347 raft_consensus.cc:388] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:49:39.405205 21347 raft_consensus.cc:482] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:49:39.405407 21347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [term 2 FOLLOWER]: Advancing to term 3
I20260810 00:49:39.409149 21347 raft_consensus.cc:504] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491fe8c474e84bd9b275042408d6adbb" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 44053 } }
I20260810 00:49:39.409639 21347 leader_election.cc:302] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 491fe8c474e84bd9b275042408d6adbb; no voters:
I20260810 00:49:39.410341 21347 leader_election.cc:288] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 00:49:39.410465 21356 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [term 3 FOLLOWER]: Leader election won for term 3
I20260810 00:49:39.411836 21356 raft_consensus.cc:686] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [term 3 LEADER]: Becoming Leader. State: Replica: 491fe8c474e84bd9b275042408d6adbb, State: Running, Role: LEADER
I20260810 00:49:39.412375 21356 consensus_queue.cc:237] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491fe8c474e84bd9b275042408d6adbb" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 44053 } }
I20260810 00:49:39.412672 21347 sys_catalog.cc:564] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:49:39.415482 21358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "491fe8c474e84bd9b275042408d6adbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491fe8c474e84bd9b275042408d6adbb" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 44053 } } }
I20260810 00:49:39.415856 21358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [sys.catalog]: This master's current role is: LEADER
I20260810 00:49:39.415477 21360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 491fe8c474e84bd9b275042408d6adbb. Latest consensus state: current_term: 3 leader_uuid: "491fe8c474e84bd9b275042408d6adbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491fe8c474e84bd9b275042408d6adbb" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 44053 } } }
I20260810 00:49:39.415973 21360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb [sys.catalog]: This master's current role is: LEADER
I20260810 00:49:39.416776 21364 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:49:39.436600 21364 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:49:39.438138 21364 catalog_manager.cc:1259] Loaded cluster ID: 8e4bca419786490cbc821bfe13a74ebc
I20260810 00:49:39.438338 21364 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:49:39.442483 21364 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:49:39.444183 21364 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb: Loaded TSK: 0
I20260810 00:49:39.444432 21364 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 491fe8c474e84bd9b275042408d6adbb: Loaded TSK: 1
F20260810 00:49:39.445813 21364 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f2ec24f19ad google::LogMessage::Fail() at ??:0
@ 0x7f2ec24f488a google::LogMessage::SendToLog() at ??:0
@ 0x7f2ec24f14de google::LogMessage::Flush() at ??:0
@ 0x7f2ec24f2a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f2ec62557d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f2ec624473e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f2ed7f201c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f2ed7f1f783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f2ed7fab2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f2ed7fab0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f2ed7ff55e9 std::function<>::operator()() at ??:0
@ 0x7f2ed7f23cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f2ed7f21c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f2ed7fa9610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f2ed7fa9439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f2ec513d345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f2ec514091c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f2ec5140799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f2ec511beaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2ec83f56db start_thread at ??:0
@ 0x7f2ec138b61f clone at ??:0
W20260810 00:49:39.760644 21351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:49:39.761026 21351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:49:39.761197 21351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:49:39.771054 21351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:49:39.771185 21351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:49:39.771433 21351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.1
I20260810 00:49:39.789978 21351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:39585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.1:35567
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.6.1
--webserver_port=46097
--tserver_master_addrs=127.28.6.62:44053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.1
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:49:39.791283 21351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:49:39.792588 21351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:49:39.805809 21429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:49:39.805879 21427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:49:39.805898 21434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:49:39.808254 21351 server_base.cc:1034] running on GCE node
I20260810 00:49:39.808830 21351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:49:39.809653 21351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:49:39.810849 21351 hybrid_clock.cc:648] HybridClock initialized: now 1786322979810816 us; error 30 us; skew 500 ppm
I20260810 00:49:39.811198 21351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:49:39 dist-test-slave-1nl6 krb5kdc[19398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322979, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:49:39.814231 21351 init.cc:377] Logged in from keytab as kudu/127.28.6.1@KRBTEST.COM (short username kudu)
I20260810 00:49:39.815766 21351 webserver.cc:460] Webserver started at http://127.28.6.1:46097/ using document root <none> and password file <none>
I20260810 00:49:39.816455 21351 fs_manager.cc:359] Metadata directory not provided
I20260810 00:49:39.816605 21351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:49:39.821853 21351 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260810 00:49:39.824731 21445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:49:39.825929 21351 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 00:49:39.826097 21351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/wal
uuid: "92f2b001cfd2415dba06994af259aabb"
format_stamp: "Formatted at 2026-08-10 00:48:53 on dist-test-slave-1nl6"
server_key: "d41800cbef261e62d0c2da089016b595"
server_key_iv: "801c02d3a8d6bf5d64dcc54e4375f125"
server_key_version: "encryptionkey@0"
I20260810 00:49:39.827109 21351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:49:39.865928 21351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:49:39.869674 21351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:49:39.869959 21351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:49:39.870889 21351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:49:39.872691 21455 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260810 00:49:39.878518 21351 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 00:49:39.878654 21351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260810 00:49:39.878793 21351 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 00:49:39.881976 21351 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 00:49:39.882086 21351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260810 00:49:39.882390 21455 tablet_bootstrap.cc:492] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb: Bootstrap starting.
I20260810 00:49:39.890618 21455 log.cc:824] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb: Log is configured to *not* fsync() on all Append() calls
I20260810 00:49:39.908509 21455 tablet_bootstrap.cc:492] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 00:49:39.909521 21455 tablet_bootstrap.cc:492] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb: Bootstrap complete.
I20260810 00:49:39.910156 21455 ts_tablet_manager.cc:1397] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb: Time spent bootstrapping tablet: real 0.028s user 0.024s sys 0.002s
I20260810 00:49:39.922008 21455 raft_consensus.cc:348] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f2b001cfd2415dba06994af259aabb" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 35567 } } peers { permanent_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34761 } } peers { permanent_uuid: "c2aa322985374b8092a1f4adb947ab42" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46009 } }
I20260810 00:49:39.923662 21455 raft_consensus.cc:729] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92f2b001cfd2415dba06994af259aabb, State: Initialized, Role: FOLLOWER
I20260810 00:49:39.923840 21351 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.1:35567
I20260810 00:49:39.923902 21644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.1:35567 every 8 connection(s)
I20260810 00:49:39.924520 21455 consensus_queue.cc:260] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f2b001cfd2415dba06994af259aabb" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 35567 } } peers { permanent_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34761 } } peers { permanent_uuid: "c2aa322985374b8092a1f4adb947ab42" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46009 } }
I20260810 00:49:39.925191 21351 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
I20260810 00:49:39.926765 21455 ts_tablet_manager.cc:1428] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb: Time spent starting tablet: real 0.016s user 0.005s sys 0.012s
I20260810 00:49:39.929939 21652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:39.928161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.1:52551)
0810 00:49:39.928912 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:39.929099 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":323,"threads_started":1}
W20260810 00:49:39.930609 21648 heartbeater.cc:644] Failed to heartbeat to 127.28.6.62:44053 (0 consecutive failures): Network error: Failed to ping master at 127.28.6.62:44053: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
I20260810 00:49:39.931106 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 21351
I20260810 00:49:39.931481 21652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:39.931136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.1:53229)
0810 00:49:39.931280 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:39.931357 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 00:49:39.931890 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.2:34761
--local_ip_for_outbound_sockets=127.28.6.2
--tserver_master_addrs=127.28.6.62:44053
--webserver_port=41307
--webserver_interface=127.28.6.2
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:39585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 00:49:39.932379 21652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:39.932013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.1:45517)
0810 00:49:39.932175 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:39.932266 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 00:49:39.932644 21648 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 00:49:40.281594 21653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:49:40.281893 21653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:49:40.282056 21653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:49:40.289146 21653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:49:40.289278 21653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:49:40.289467 21653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.2
I20260810 00:49:40.298931 21653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:39585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.2:34761
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.6.2
--webserver_port=41307
--tserver_master_addrs=127.28.6.62:44053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.2
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:49:40.300186 21653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:49:40.301486 21653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:49:40.314627 21667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:49:40.314716 21664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:49:40.314682 21665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:49:40.315629 21653 server_base.cc:1034] running on GCE node
I20260810 00:49:40.316174 21653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:49:40.317000 21653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:49:40.318228 21653 hybrid_clock.cc:648] HybridClock initialized: now 1786322980318193 us; error 60 us; skew 500 ppm
I20260810 00:49:40.318540 21653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:49:40 dist-test-slave-1nl6 krb5kdc[19398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322980, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:49:40.321645 21653 init.cc:377] Logged in from keytab as kudu/127.28.6.2@KRBTEST.COM (short username kudu)
I20260810 00:49:40.323156 21653 webserver.cc:460] Webserver started at http://127.28.6.2:41307/ using document root <none> and password file <none>
I20260810 00:49:40.323827 21653 fs_manager.cc:359] Metadata directory not provided
I20260810 00:49:40.323979 21653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:49:40.329169 21653 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260810 00:49:40.332120 21676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:49:40.333473 21653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 00:49:40.333618 21653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/wal
uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea"
format_stamp: "Formatted at 2026-08-10 00:48:54 on dist-test-slave-1nl6"
server_key: "4930f94ecea10c721337ac33af729447"
server_key_iv: "e875c1dc1d9b0dea9f7e634fd9211dd5"
server_key_version: "encryptionkey@0"
I20260810 00:49:40.334548 21653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:49:40.355365 21653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:49:40.359213 21653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:49:40.359442 21653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:49:40.360430 21653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:49:40.361702 21653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:49:40.361789 21653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:49:40.361872 21653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:49:40.361939 21653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:49:40.397220 21653 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.2:34761
I20260810 00:49:40.397280 21863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.2:34761 every 8 connection(s)
I20260810 00:49:40.398610 21653 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
I20260810 00:49:40.401024 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 21653
I20260810 00:49:40.401625 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.3:46009
--local_ip_for_outbound_sockets=127.28.6.3
--tserver_master_addrs=127.28.6.62:44053
--webserver_port=39153
--webserver_interface=127.28.6.3
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:39585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 00:49:40.403966 21866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:40.401613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.2:49721)
0810 00:49:40.402489 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:40.402758 (+ 269us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":316,"threads_started":1}
W20260810 00:49:40.404707 21864 heartbeater.cc:644] Failed to heartbeat to 127.28.6.62:44053 (0 consecutive failures): Network error: Failed to ping master at 127.28.6.62:44053: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
I20260810 00:49:40.405647 21866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:40.405225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.2:51373)
0810 00:49:40.405408 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:40.405498 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260810 00:49:40.406587 21866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:40.406175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.2:55867)
0810 00:49:40.406340 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:40.406439 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 00:49:40.406864 21864 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 00:49:40.737715 21867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:49:40.737982 21867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:49:40.738168 21867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:49:40.745489 21867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:49:40.745594 21867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:49:40.745801 21867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.3
I20260810 00:49:40.755363 21867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:39585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.3:46009
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.6.3
--webserver_port=39153
--tserver_master_addrs=127.28.6.62:44053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.3
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:49:40.756551 21867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:49:40.757838 21867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:49:40.770967 21876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:49:40.771070 21875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:49:40.771297 21878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:49:40.772060 21867 server_base.cc:1034] running on GCE node
I20260810 00:49:40.772637 21867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:49:40.773514 21867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:49:40.774714 21867 hybrid_clock.cc:648] HybridClock initialized: now 1786322980774675 us; error 32 us; skew 500 ppm
I20260810 00:49:40.775074 21867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:49:40 dist-test-slave-1nl6 krb5kdc[19398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322980, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:49:40.778226 21867 init.cc:377] Logged in from keytab as kudu/127.28.6.3@KRBTEST.COM (short username kudu)
I20260810 00:49:40.779696 21867 webserver.cc:460] Webserver started at http://127.28.6.3:39153/ using document root <none> and password file <none>
I20260810 00:49:40.780294 21867 fs_manager.cc:359] Metadata directory not provided
I20260810 00:49:40.780404 21867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:49:40.785615 21867 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260810 00:49:40.788129 21889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:49:40.789376 21867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 00:49:40.789564 21867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/wal
uuid: "c2aa322985374b8092a1f4adb947ab42"
format_stamp: "Formatted at 2026-08-10 00:48:55 on dist-test-slave-1nl6"
server_key: "182ae569985ac8b68141a5764dd00b01"
server_key_iv: "b7fce0030639b16e31f577301e6c7789"
server_key_version: "encryptionkey@0"
I20260810 00:49:40.790486 21867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:49:40.814404 21867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:49:40.818236 21867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:49:40.818511 21867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:49:40.819499 21867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:49:40.820847 21867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:49:40.820958 21867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:49:40.821094 21867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:49:40.821180 21867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:49:40.854885 21867 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.3:46009
I20260810 00:49:40.855051 22083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.3:46009 every 8 connection(s)
I20260810 00:49:40.856364 21867 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
I20260810 00:49:40.860922 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 21867
I20260810 00:49:40.861307 22087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:40.859260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.3:42165)
0810 00:49:40.859954 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:40.860158 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":263,"threads_started":1}
W20260810 00:49:40.861966 22084 heartbeater.cc:644] Failed to heartbeat to 127.28.6.62:44053 (0 consecutive failures): Network error: Failed to ping master at 127.28.6.62:44053: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
I20260810 00:49:40.862883 22087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:40.862514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.3:38917)
0810 00:49:40.862688 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:40.862766 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 00:49:40.863853 22087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:40.863557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.3:60783)
0810 00:49:40.863681 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:40.863751 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 00:49:40.864147 22084 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 00:49:40.933820 22090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:40.933076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.1:33211)
0810 00:49:40.933504 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:40.933594 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":207,"threads_started":1}
I20260810 00:49:41.319769 22091 raft_consensus.cc:482] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:49:41.320050 22091 raft_consensus.cc:504] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f2b001cfd2415dba06994af259aabb" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 35567 } } peers { permanent_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34761 } } peers { permanent_uuid: "c2aa322985374b8092a1f4adb947ab42" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46009 } }
I20260810 00:49:41.321619 22091 leader_election.cc:288] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92c7bf0b0c8a4ace9fed6b4576ddc4ea (127.28.6.2:34761), c2aa322985374b8092a1f4adb947ab42 (127.28.6.3:46009)
Aug 10 00:49:41 dist-test-slave-1nl6 krb5kdc[19398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322979, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for kudu/127.28.6.3@KRBTEST.COM
Aug 10 00:49:41 dist-test-slave-1nl6 krb5kdc[19398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322979, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for kudu/127.28.6.2@KRBTEST.COM
I20260810 00:49:41.345005 22090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:41.321905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:34761 (local address 127.28.6.1:37471)
0810 00:49:41.322621 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:41.322679 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 00:49:41.323385 (+ 706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:49:41.327133 (+ 3748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:49:41.327143 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:49:41.328032 (+ 889us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:49:41.328781 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:49:41.328800 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:49:41.330465 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:49:41.330469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:49:41.332163 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:49:41.332180 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:49:41.334278 (+ 2098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:49:41.334285 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:49:41.334513 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:41.335191 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:49:41.335214 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:49:41.337613 (+ 2399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:49:41.341134 (+ 3521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:49:41.341141 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:49:41.341147 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:49:41.341648 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:49:41.342087 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:49:41.342097 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:49:41.342101 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:49:41.342278 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:49:41.343100 (+ 822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:49:41.343105 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:49:41.343516 (+ 411us) client_negotiation.cc:769] Sending connection context
0810 00:49:41.343858 (+ 342us) client_negotiation.cc:240] Negotiation successful
0810 00:49:41.344662 (+ 804us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":397,"spinlock_wait_cycles":39808}
I20260810 00:49:41.345029 22092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:41.322304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:46009 (local address 127.28.6.1:56143)
0810 00:49:41.324833 (+ 2529us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:41.324864 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0810 00:49:41.324997 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:49:41.327313 (+ 2316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:49:41.327318 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:49:41.328029 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:49:41.328782 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:49:41.328801 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:49:41.330471 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:49:41.330475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:49:41.332163 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:49:41.332180 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:49:41.333107 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:49:41.333112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:49:41.333311 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:41.334041 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:49:41.334064 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:49:41.337139 (+ 3075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:49:41.341083 (+ 3944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:49:41.341092 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:49:41.341106 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:49:41.341595 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:49:41.342041 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:49:41.342046 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:49:41.342049 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:49:41.342271 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:49:41.343096 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:49:41.343104 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:49:41.343519 (+ 415us) client_negotiation.cc:769] Sending connection context
0810 00:49:41.343858 (+ 339us) client_negotiation.cc:240] Negotiation successful
0810 00:49:41.344489 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2376,"mutex_wait_us":176,"spinlock_wait_cycles":9216,"thread_start_us":2206,"threads_started":1}
I20260810 00:49:41.345324 22093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:41.322174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:37471 (local address 127.28.6.2:34761)
0810 00:49:41.324583 (+ 2409us) server_negotiation.cc:206] Beginning negotiation
0810 00:49:41.324597 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 00:49:41.324640 (+ 43us) server_negotiation.cc:407] Connection header received
0810 00:49:41.326294 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:49:41.326303 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:49:41.326654 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:49:41.326966 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:49:41.329022 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:41.330261 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:41.333252 (+ 2991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:41.334064 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:41.334238 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:41.337905 (+ 3667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:49:41.337915 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:49:41.337932 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:49:41.337986 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:49:41.340993 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:49:41.341802 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:49:41.341808 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:49:41.341813 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:49:41.341967 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:49:41.342402 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:49:41.342413 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:49:41.342415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:49:41.342958 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:49:41.343157 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0810 00:49:41.344489 (+ 1332us) server_negotiation.cc:299] Negotiation successful
0810 00:49:41.344913 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2187,"thread_start_us":2068,"threads_started":1}
I20260810 00:49:41.346199 22095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:41.322867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:56143 (local address 127.28.6.3:46009)
0810 00:49:41.324299 (+ 1432us) server_negotiation.cc:206] Beginning negotiation
0810 00:49:41.324314 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 00:49:41.324957 (+ 643us) server_negotiation.cc:407] Connection header received
0810 00:49:41.326579 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:49:41.326587 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:49:41.326990 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:49:41.327331 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:49:41.328976 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:41.330275 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:41.332300 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:41.332944 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:41.333078 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:41.337489 (+ 4411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:49:41.337503 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:49:41.337521 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:49:41.337575 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:49:41.340935 (+ 3360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:49:41.341746 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:49:41.341755 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:49:41.341767 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:49:41.341922 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:49:41.342401 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:49:41.342406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:49:41.342410 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:49:41.342956 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:49:41.343157 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0810 00:49:41.345643 (+ 2486us) server_negotiation.cc:299] Negotiation successful
0810 00:49:41.345924 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1198,"thread_start_us":994,"threads_started":1}
W20260810 00:49:41.348589 21452 leader_election.cc:341] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 92c7bf0b0c8a4ace9fed6b4576ddc4ea (127.28.6.2:34761): Not found: Tablet not found: 5e10f926d13b464eb954b3c168770d1f
W20260810 00:49:41.349005 21452 leader_election.cc:341] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c2aa322985374b8092a1f4adb947ab42 (127.28.6.3:46009): Not found: Tablet not found: 5e10f926d13b464eb954b3c168770d1f
I20260810 00:49:41.349182 21452 leader_election.cc:302] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92f2b001cfd2415dba06994af259aabb; no voters: 92c7bf0b0c8a4ace9fed6b4576ddc4ea, c2aa322985374b8092a1f4adb947ab42
I20260810 00:49:41.349643 22091 raft_consensus.cc:2726] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 00:49:41.408233 22099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:41.407365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.2:54025)
0810 00:49:41.407828 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:41.407962 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":226,"threads_started":1}
I20260810 00:49:41.865413 22101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:41.864700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.3:46827)
0810 00:49:41.865129 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:41.865223 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":248,"threads_started":1}
I20260810 00:49:41.935355 22103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:41.934665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.1:44577)
0810 00:49:41.935095 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:41.935169 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":225,"threads_started":1}
I20260810 00:49:42.409653 22106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:42.409024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.2:43495)
0810 00:49:42.409429 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:42.409503 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":223,"threads_started":1}
I20260810 00:49:42.866950 22109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:42.866252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.3:47413)
0810 00:49:42.866670 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:42.866753 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":222,"threads_started":1}
I20260810 00:49:42.936837 22111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:42.936145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.1:42485)
0810 00:49:42.936586 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:42.936658 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":220,"threads_started":1}
I20260810 00:49:43.220388 22114 raft_consensus.cc:482] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:49:43.220580 22114 raft_consensus.cc:504] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f2b001cfd2415dba06994af259aabb" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 35567 } } peers { permanent_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34761 } } peers { permanent_uuid: "c2aa322985374b8092a1f4adb947ab42" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46009 } }
I20260810 00:49:43.221160 22114 leader_election.cc:288] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92c7bf0b0c8a4ace9fed6b4576ddc4ea (127.28.6.2:34761), c2aa322985374b8092a1f4adb947ab42 (127.28.6.3:46009)
W20260810 00:49:43.222259 21452 leader_election.cc:341] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 92c7bf0b0c8a4ace9fed6b4576ddc4ea (127.28.6.2:34761): Not found: Tablet not found: 5e10f926d13b464eb954b3c168770d1f
W20260810 00:49:43.222520 21452 leader_election.cc:341] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c2aa322985374b8092a1f4adb947ab42 (127.28.6.3:46009): Not found: Tablet not found: 5e10f926d13b464eb954b3c168770d1f
I20260810 00:49:43.222632 21452 leader_election.cc:302] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92f2b001cfd2415dba06994af259aabb; no voters: 92c7bf0b0c8a4ace9fed6b4576ddc4ea, c2aa322985374b8092a1f4adb947ab42
I20260810 00:49:43.222959 22114 raft_consensus.cc:2726] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 00:49:43.411170 22115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:43.410436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.2:37943)
0810 00:49:43.410868 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:43.411011 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":232,"threads_started":1}
I20260810 00:49:43.868480 22117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:43.867776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.3:44215)
0810 00:49:43.868227 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:43.868313 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":247,"threads_started":1}
I20260810 00:49:43.938450 22118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:43.937681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.1:50609)
0810 00:49:43.938204 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:43.938285 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":294,"threads_started":1}
I20260810 00:49:44.412681 22119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:44.411981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.2:35301)
0810 00:49:44.412412 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:44.412493 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":216,"threads_started":1}
I20260810 00:49:44.869983 22120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:44.869265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.3:39173)
0810 00:49:44.869695 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:44.869769 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":205,"threads_started":1}
I20260810 00:49:44.940093 22122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:44.939381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.1:56761)
0810 00:49:44.939838 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:44.939909 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":225,"threads_started":1}
I20260810 00:49:45.002458 22123 raft_consensus.cc:482] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:49:45.002696 22123 raft_consensus.cc:504] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f2b001cfd2415dba06994af259aabb" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 35567 } } peers { permanent_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34761 } } peers { permanent_uuid: "c2aa322985374b8092a1f4adb947ab42" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46009 } }
I20260810 00:49:45.003474 22123 leader_election.cc:288] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92c7bf0b0c8a4ace9fed6b4576ddc4ea (127.28.6.2:34761), c2aa322985374b8092a1f4adb947ab42 (127.28.6.3:46009)
W20260810 00:49:45.004474 21452 leader_election.cc:341] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 92c7bf0b0c8a4ace9fed6b4576ddc4ea (127.28.6.2:34761): Not found: Tablet not found: 5e10f926d13b464eb954b3c168770d1f
W20260810 00:49:45.004710 21452 leader_election.cc:341] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c2aa322985374b8092a1f4adb947ab42 (127.28.6.3:46009): Not found: Tablet not found: 5e10f926d13b464eb954b3c168770d1f
I20260810 00:49:45.004874 21452 leader_election.cc:302] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92f2b001cfd2415dba06994af259aabb; no voters: 92c7bf0b0c8a4ace9fed6b4576ddc4ea, c2aa322985374b8092a1f4adb947ab42
I20260810 00:49:45.005124 22123 raft_consensus.cc:2726] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 00:49:45.414383 22126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:45.413659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.2:54221)
0810 00:49:45.414140 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:45.414223 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":285,"threads_started":1}
I20260810 00:49:45.871850 22127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:45.871074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.3:40451)
0810 00:49:45.871616 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:45.871700 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":330,"threads_started":1}
I20260810 00:49:45.941569 22130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:45.940924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.1:38729)
0810 00:49:45.941334 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:45.941419 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":224,"threads_started":1}
I20260810 00:49:46.416241 22131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:46.415287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.2:46975)
0810 00:49:46.415868 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:46.415989 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":272,"threads_started":1}
I20260810 00:49:46.873382 22132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:46.872704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.3:55979)
0810 00:49:46.873131 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:46.873206 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":234,"threads_started":1}
I20260810 00:49:46.925381 22133 raft_consensus.cc:482] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:49:46.925592 22133 raft_consensus.cc:504] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f2b001cfd2415dba06994af259aabb" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 35567 } } peers { permanent_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34761 } } peers { permanent_uuid: "c2aa322985374b8092a1f4adb947ab42" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46009 } }
I20260810 00:49:46.926249 22133 leader_election.cc:288] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92c7bf0b0c8a4ace9fed6b4576ddc4ea (127.28.6.2:34761), c2aa322985374b8092a1f4adb947ab42 (127.28.6.3:46009)
W20260810 00:49:46.927325 21452 leader_election.cc:341] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 92c7bf0b0c8a4ace9fed6b4576ddc4ea (127.28.6.2:34761): Not found: Tablet not found: 5e10f926d13b464eb954b3c168770d1f
W20260810 00:49:46.927591 21452 leader_election.cc:341] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c2aa322985374b8092a1f4adb947ab42 (127.28.6.3:46009): Not found: Tablet not found: 5e10f926d13b464eb954b3c168770d1f
I20260810 00:49:46.927713 21452 leader_election.cc:302] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92f2b001cfd2415dba06994af259aabb; no voters: 92c7bf0b0c8a4ace9fed6b4576ddc4ea, c2aa322985374b8092a1f4adb947ab42
I20260810 00:49:46.927975 22133 raft_consensus.cc:2726] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 00:49:46.943264 22135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:46.942452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.1:32883)
0810 00:49:46.943000 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:46.943082 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":285,"threads_started":1}
I20260810 00:49:47.417902 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:47.417159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.2:58837)
0810 00:49:47.417634 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:47.417710 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":216,"threads_started":1}
I20260810 00:49:47.875128 22137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:47.874290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.3:58197)
0810 00:49:47.874780 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:47.874856 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":228,"threads_started":1}
I20260810 00:49:47.944729 22139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:47.944097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.1:60769)
0810 00:49:47.944506 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:47.944588 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":231,"threads_started":1}
I20260810 00:49:48.419803 22140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:48.419010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.2:46285)
0810 00:49:48.419461 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:48.419560 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":237,"threads_started":1}
I20260810 00:49:48.876822 22141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:48.876035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.3:59441)
0810 00:49:48.876571 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:48.876647 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":293,"threads_started":1}
I20260810 00:49:48.946344 22143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:48.945626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.1:35865)
0810 00:49:48.946076 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:48.946159 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":254,"threads_started":1}
I20260810 00:49:49.421396 22144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:49.420700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.2:60083)
0810 00:49:49.421153 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:49.421228 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":220,"threads_started":1}
I20260810 00:49:49.878424 22146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:49.877733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.3:36931)
0810 00:49:49.878180 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:49.878270 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":213,"threads_started":1}
I20260810 00:49:49.947942 22149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:49.947290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.1:53541)
0810 00:49:49.947708 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:49.947788 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":211,"threads_started":1}
I20260810 00:49:50.423143 22150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:50.422369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.2:38095)
0810 00:49:50.422805 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:50.422946 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":245,"threads_started":1}
I20260810 00:49:50.880034 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:50.879394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.3:56493)
0810 00:49:50.879793 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:50.879865 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":221,"threads_started":1}
I20260810 00:49:50.949577 22153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:50.948753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.1:39535)
0810 00:49:50.949285 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:50.949369 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":338,"threads_started":1}
I20260810 00:49:51.354053 22155 raft_consensus.cc:482] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:49:51.354275 22155 raft_consensus.cc:504] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f2b001cfd2415dba06994af259aabb" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 35567 } } peers { permanent_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34761 } } peers { permanent_uuid: "c2aa322985374b8092a1f4adb947ab42" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46009 } }
I20260810 00:49:51.354945 22155 leader_election.cc:288] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92c7bf0b0c8a4ace9fed6b4576ddc4ea (127.28.6.2:34761), c2aa322985374b8092a1f4adb947ab42 (127.28.6.3:46009)
W20260810 00:49:51.356004 21452 leader_election.cc:341] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 92c7bf0b0c8a4ace9fed6b4576ddc4ea (127.28.6.2:34761): Not found: Tablet not found: 5e10f926d13b464eb954b3c168770d1f
W20260810 00:49:51.356263 21452 leader_election.cc:341] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c2aa322985374b8092a1f4adb947ab42 (127.28.6.3:46009): Not found: Tablet not found: 5e10f926d13b464eb954b3c168770d1f
I20260810 00:49:51.356406 21452 leader_election.cc:302] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92f2b001cfd2415dba06994af259aabb; no voters: 92c7bf0b0c8a4ace9fed6b4576ddc4ea, c2aa322985374b8092a1f4adb947ab42
I20260810 00:49:51.356685 22155 raft_consensus.cc:2726] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 00:49:51.424786 22157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:51.424009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.2:52651)
0810 00:49:51.424540 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:51.424625 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":270,"threads_started":1}
I20260810 00:49:51.881771 22158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:51.880968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.3:36317)
0810 00:49:51.881543 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:51.881622 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":327,"threads_started":1}
I20260810 00:49:51.951325 22159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:51.950473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.1:50899)
0810 00:49:51.951033 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:51.951131 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":290,"threads_started":1}
I20260810 00:49:52.426422 22160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:52.425749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.2:43665)
0810 00:49:52.426176 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:52.426254 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":221,"threads_started":1}
I20260810 00:49:52.883474 22161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:52.882673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.3:59899)
0810 00:49:52.883210 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:52.883292 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":330,"threads_started":1}
I20260810 00:49:52.952975 22163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:52.952241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.1:35757)
0810 00:49:52.952710 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:52.952793 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":255,"threads_started":1}
I20260810 00:49:53.428162 22164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:53.427429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.2:59095)
0810 00:49:53.427915 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:53.427992 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":249,"threads_started":1}
I20260810 00:49:53.885236 22165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:53.884456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.3:60139)
0810 00:49:53.885013 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:53.885088 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":293,"threads_started":1}
I20260810 00:49:53.954442 22168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:53.953771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.1:51091)
0810 00:49:53.954192 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:53.954267 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":221,"threads_started":1}
I20260810 00:49:54.430071 22169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:54.429174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.2:37785)
0810 00:49:54.429708 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:54.429792 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":302,"threads_started":1}
I20260810 00:49:54.607225 22171 raft_consensus.cc:482] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:49:54.607436 22171 raft_consensus.cc:504] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92f2b001cfd2415dba06994af259aabb" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 35567 } } peers { permanent_uuid: "92c7bf0b0c8a4ace9fed6b4576ddc4ea" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 34761 } } peers { permanent_uuid: "c2aa322985374b8092a1f4adb947ab42" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 46009 } }
I20260810 00:49:54.608031 22171 leader_election.cc:288] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92c7bf0b0c8a4ace9fed6b4576ddc4ea (127.28.6.2:34761), c2aa322985374b8092a1f4adb947ab42 (127.28.6.3:46009)
W20260810 00:49:54.609030 21452 leader_election.cc:341] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 92c7bf0b0c8a4ace9fed6b4576ddc4ea (127.28.6.2:34761): Not found: Tablet not found: 5e10f926d13b464eb954b3c168770d1f
W20260810 00:49:54.609262 21452 leader_election.cc:341] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c2aa322985374b8092a1f4adb947ab42 (127.28.6.3:46009): Not found: Tablet not found: 5e10f926d13b464eb954b3c168770d1f
I20260810 00:49:54.609350 21452 leader_election.cc:302] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92f2b001cfd2415dba06994af259aabb; no voters: 92c7bf0b0c8a4ace9fed6b4576ddc4ea, c2aa322985374b8092a1f4adb947ab42
I20260810 00:49:54.609573 22171 raft_consensus.cc:2726] T 5e10f926d13b464eb954b3c168770d1f P 92f2b001cfd2415dba06994af259aabb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 00:49:54.887061 22174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:54.886075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.3:41597)
0810 00:49:54.886721 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:54.886801 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":326,"threads_started":1}
I20260810 00:49:54.956300 22175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:54.955553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.1:41273)
0810 00:49:54.956035 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:54.956110 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":219,"threads_started":1}
I20260810 00:49:55.431866 22177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:55.431175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.2:35501)
0810 00:49:55.431605 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:55.431700 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":220,"threads_started":1}
I20260810 00:49:55.867626 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 21351
I20260810 00:49:55.888859 22179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:55.888119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:44053 (local address 127.28.6.3:34087)
0810 00:49:55.888551 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:55.888639 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":225,"threads_started":1}
I20260810 00:49:55.899289 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:55.868835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43442 (local address 127.28.6.1:35567)
0810 00:49:55.869422 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0810 00:49:55.869429 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:49:55.869456 (+ 27us) server_negotiation.cc:407] Connection header received
0810 00:49:55.869528 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:49:55.869535 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:49:55.869752 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:49:55.869939 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:49:55.871514 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:55.872539 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:55.873744 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:55.874297 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:55.874404 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:55.893775 (+ 19371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:49:55.893791 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:49:55.893807 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:49:55.893844 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:49:55.896761 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:49:55.897460 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:49:55.897467 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:49:55.897470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:49:55.897601 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:49:55.897928 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:49:55.897931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:49:55.897933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:49:55.898277 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:49:55.898518 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0810 00:49:55.898749 (+ 231us) server_negotiation.cc:299] Negotiation successful
0810 00:49:55.899041 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":300,"threads_started":1}
I20260810 00:49:55.900590 21635 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:49:55.988790 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 21351
I20260810 00:49:56.017432 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 21653
I20260810 00:49:56.031395 22184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:56.018234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50170 (local address 127.28.6.2:34761)
0810 00:49:56.018757 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0810 00:49:56.018761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:49:56.018778 (+ 17us) server_negotiation.cc:407] Connection header received
0810 00:49:56.018847 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:49:56.018851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:49:56.018966 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:49:56.019063 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:49:56.020380 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:56.021195 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:56.022502 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:56.023096 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:56.023196 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:56.025987 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:49:56.025994 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:49:56.026002 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:49:56.026062 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:49:56.029016 (+ 2954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:49:56.029668 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:49:56.029671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:49:56.029674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:49:56.029789 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:49:56.030119 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:49:56.030122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:49:56.030124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:49:56.030369 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:49:56.030559 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0810 00:49:56.030850 (+ 291us) server_negotiation.cc:299] Negotiation successful
0810 00:49:56.031089 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":304,"threads_started":1}
I20260810 00:49:56.032203 21857 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:49:56.117864 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 21653
I20260810 00:49:56.148880 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 21867
I20260810 00:49:56.162715 22190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:56.149647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58504 (local address 127.28.6.3:46009)
0810 00:49:56.150171 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0810 00:49:56.150176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:49:56.150192 (+ 16us) server_negotiation.cc:407] Connection header received
0810 00:49:56.150252 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:49:56.150255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:49:56.150330 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:49:56.150432 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:49:56.151895 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:56.152754 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:56.153929 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:56.154616 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:56.154763 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:56.157448 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:49:56.157453 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:49:56.157457 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:49:56.157490 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:49:56.160293 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:49:56.160984 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:49:56.160988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:49:56.160990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:49:56.161127 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:49:56.161452 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:49:56.161455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:49:56.161458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:49:56.161698 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:49:56.161897 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0810 00:49:56.162148 (+ 251us) server_negotiation.cc:299] Negotiation successful
0810 00:49:56.162302 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":274,"threads_started":1}
I20260810 00:49:56.163445 22076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:49:56.246336 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 21867
2026-08-10T00:49:56Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64817 ms)
[----------] 30 tests from SecurityITest (308989 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:49:56 dist-test-slave-1nl6 krb5kdc[22197](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:49:56 dist-test-slave-1nl6 krb5kdc[22197](info): set up 2 sockets
Aug 10 00:49:56 dist-test-slave-1nl6 krb5kdc[22197](info): commencing operation
krb5kdc: starting...
W20260810 00:49:57.564918 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.270s user 0.000s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:49:57 dist-test-slave-1nl6 krb5kdc[22197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:49:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:49:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.6.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
I20260810 00:49:57.844076 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:39453
--webserver_interface=127.28.6.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:37241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:39453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:49:58.180357 22222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:49:58.180604 22222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:49:58.180692 22222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:49:58.187695 22222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:49:58.187785 22222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:49:58.187901 22222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:49:58.187995 22222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:49:58.188076 22222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:49:58.197381 22222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:37241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:39453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:39453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:49:58.198535 22222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:49:58.199841 22222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:49:58.211941 22229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:49:58.212021 22230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:49:58.212018 22233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:49:58.212990 22222 server_base.cc:1034] running on GCE node
I20260810 00:49:58.213797 22222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:49:58.215149 22222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:49:58.216388 22222 hybrid_clock.cc:648] HybridClock initialized: now 1786322998216351 us; error 45 us; skew 500 ppm
I20260810 00:49:58.216729 22222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:49:58 dist-test-slave-1nl6 krb5kdc[22197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322998, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:49:58.219894 22222 init.cc:377] Logged in from keytab as kudu/127.28.6.62@KRBTEST.COM (short username kudu)
I20260810 00:49:58.221369 22222 webserver.cc:460] Webserver started at http://127.28.6.62:36735/ using document root <none> and password file <none>
I20260810 00:49:58.222033 22222 fs_manager.cc:359] Metadata directory not provided
I20260810 00:49:58.222175 22222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:49:58.222483 22222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:49:58.225405 22222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "21c24236ed894a9495841b718ebda2c5"
format_stamp: "Formatted at 2026-08-10 00:49:58 on dist-test-slave-1nl6"
server_key: "2126db5febe422f271096db83a757a32"
server_key_iv: "026d015a3277984d348c4d3a78aac006"
server_key_version: "encryptionkey@0"
I20260810 00:49:58.226150 22222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "21c24236ed894a9495841b718ebda2c5"
format_stamp: "Formatted at 2026-08-10 00:49:58 on dist-test-slave-1nl6"
server_key: "2126db5febe422f271096db83a757a32"
server_key_iv: "026d015a3277984d348c4d3a78aac006"
server_key_version: "encryptionkey@0"
I20260810 00:49:58.231019 22222 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.002s
I20260810 00:49:58.234243 22245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:49:58.235460 22222 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260810 00:49:58.235638 22222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "21c24236ed894a9495841b718ebda2c5"
format_stamp: "Formatted at 2026-08-10 00:49:58 on dist-test-slave-1nl6"
server_key: "2126db5febe422f271096db83a757a32"
server_key_iv: "026d015a3277984d348c4d3a78aac006"
server_key_version: "encryptionkey@0"
I20260810 00:49:58.235827 22222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:49:58.263177 22222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:49:58.266847 22222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:49:58.267143 22222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:49:58.285408 22222 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:39453
I20260810 00:49:58.285454 22331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:39453 every 8 connection(s)
I20260810 00:49:58.286695 22222 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:49:58.290228 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 22222
I20260810 00:49:58.290477 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:49:58.290596 22332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:49:58.291416 28696 external_mini_cluster.cc:1428] Setting key 0b0cf175c1ce08d85b234792105f5018
I20260810 00:49:58.305598 22332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21c24236ed894a9495841b718ebda2c5: Bootstrap starting.
Aug 10 00:49:58 dist-test-slave-1nl6 krb5kdc[22197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:49:58.308729 22332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21c24236ed894a9495841b718ebda2c5: Neither blocks nor log segments found. Creating new log.
I20260810 00:49:58.309768 22332 log.cc:824] T 00000000000000000000000000000000 P 21c24236ed894a9495841b718ebda2c5: Log is configured to *not* fsync() on all Append() calls
I20260810 00:49:58.312467 22332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21c24236ed894a9495841b718ebda2c5: No bootstrap required, opened a new log
I20260810 00:49:58.314643 22335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:58.293544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40936 (local address 127.28.6.62:39453)
0810 00:49:58.294381 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0810 00:49:58.294392 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 00:49:58.294432 (+ 40us) server_negotiation.cc:407] Connection header received
0810 00:49:58.295966 (+ 1534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:49:58.295987 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:49:58.296329 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:49:58.296632 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:49:58.298510 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:58.299813 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:58.301742 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:58.302408 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:58.302535 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:58.307306 (+ 4771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:49:58.307313 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:49:58.307325 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:49:58.307362 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:49:58.310330 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:49:58.311099 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:49:58.311105 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:49:58.311111 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:49:58.311271 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:49:58.311673 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:49:58.311680 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:49:58.311688 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:49:58.312139 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:49:58.312459 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0810 00:49:58.312928 (+ 469us) server_negotiation.cc:299] Negotiation successful
0810 00:49:58.313373 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":388,"threads_started":1}
I20260810 00:49:58.320519 22332 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21c24236ed894a9495841b718ebda2c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c24236ed894a9495841b718ebda2c5" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39453 } }
I20260810 00:49:58.320830 22332 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21c24236ed894a9495841b718ebda2c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:49:58.320910 22332 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21c24236ed894a9495841b718ebda2c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21c24236ed894a9495841b718ebda2c5, State: Initialized, Role: FOLLOWER
I20260810 00:49:58.321365 22332 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21c24236ed894a9495841b718ebda2c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c24236ed894a9495841b718ebda2c5" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39453 } }
I20260810 00:49:58.321563 22332 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21c24236ed894a9495841b718ebda2c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:49:58.321656 22332 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21c24236ed894a9495841b718ebda2c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:49:58.321771 22332 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21c24236ed894a9495841b718ebda2c5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:49:58.323810 22332 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21c24236ed894a9495841b718ebda2c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c24236ed894a9495841b718ebda2c5" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39453 } }
I20260810 00:49:58.324237 22332 leader_election.cc:302] T 00000000000000000000000000000000 P 21c24236ed894a9495841b718ebda2c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21c24236ed894a9495841b718ebda2c5; no voters:
I20260810 00:49:58.324750 22332 leader_election.cc:288] T 00000000000000000000000000000000 P 21c24236ed894a9495841b718ebda2c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:49:58.324903 22339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21c24236ed894a9495841b718ebda2c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:49:58.325599 22339 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21c24236ed894a9495841b718ebda2c5 [term 1 LEADER]: Becoming Leader. State: Replica: 21c24236ed894a9495841b718ebda2c5, State: Running, Role: LEADER
I20260810 00:49:58.326152 22339 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21c24236ed894a9495841b718ebda2c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c24236ed894a9495841b718ebda2c5" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39453 } }
I20260810 00:49:58.326752 22332 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21c24236ed894a9495841b718ebda2c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:49:58.328785 22340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21c24236ed894a9495841b718ebda2c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21c24236ed894a9495841b718ebda2c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c24236ed894a9495841b718ebda2c5" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39453 } } }
I20260810 00:49:58.329110 22340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21c24236ed894a9495841b718ebda2c5 [sys.catalog]: This master's current role is: LEADER
I20260810 00:49:58.329602 22341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21c24236ed894a9495841b718ebda2c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21c24236ed894a9495841b718ebda2c5. Latest consensus state: current_term: 1 leader_uuid: "21c24236ed894a9495841b718ebda2c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c24236ed894a9495841b718ebda2c5" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39453 } } }
I20260810 00:49:58.329924 22341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21c24236ed894a9495841b718ebda2c5 [sys.catalog]: This master's current role is: LEADER
I20260810 00:49:58.331581 22349 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:49:58.336711 22349 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:49:58.345084 22349 catalog_manager.cc:1347] Generated new cluster ID: 8eaa293c4456400da7c168f84a955be3
I20260810 00:49:58.345182 22349 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:49:58.380192 22349 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:49:58.381542 22349 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:49:58.389119 22349 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21c24236ed894a9495841b718ebda2c5: Generated new TSK 0
I20260810 00:49:58.389904 22349 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.6.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab.
Entry for principal kudu/127.28.6.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab.
I20260810 00:49:58.464218 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.1:0
--local_ip_for_outbound_sockets=127.28.6.1
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:39453
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:37241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:49:58.820770 22377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:49:58.821026 22377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:49:58.821116 22377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:49:58.828166 22377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:49:58.828261 22377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:49:58.828387 22377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.1
I20260810 00:49:58.837967 22377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:37241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:39453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.1
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:49:58.839159 22377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:49:58.840503 22377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:49:58.853259 22384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:49:58.853356 22383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:49:58.853609 22387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:49:58.854293 22377 server_base.cc:1034] running on GCE node
I20260810 00:49:58.854803 22377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:49:58.855747 22377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:49:58.856977 22377 hybrid_clock.cc:648] HybridClock initialized: now 1786322998856934 us; error 35 us; skew 500 ppm
I20260810 00:49:58.857314 22377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:49:58 dist-test-slave-1nl6 krb5kdc[22197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322998, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:49:58.860666 22377 init.cc:377] Logged in from keytab as kudu/127.28.6.1@KRBTEST.COM (short username kudu)
I20260810 00:49:58.862138 22377 webserver.cc:460] Webserver started at http://127.28.6.1:46329/ using document root <none> and password file <none>
I20260810 00:49:58.862807 22377 fs_manager.cc:359] Metadata directory not provided
I20260810 00:49:58.862996 22377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:49:58.863315 22377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:49:58.866111 22377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-0/data/instance:
uuid: "8128a63efa534cf29e1d5e22750d38ea"
format_stamp: "Formatted at 2026-08-10 00:49:58 on dist-test-slave-1nl6"
server_key: "fa415bf75b5885bab9925d4645f50930"
server_key_iv: "2548b9a9f290ae227d66d60c7bec81d4"
server_key_version: "encryptionkey@0"
I20260810 00:49:58.866890 22377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance:
uuid: "8128a63efa534cf29e1d5e22750d38ea"
format_stamp: "Formatted at 2026-08-10 00:49:58 on dist-test-slave-1nl6"
server_key: "fa415bf75b5885bab9925d4645f50930"
server_key_iv: "2548b9a9f290ae227d66d60c7bec81d4"
server_key_version: "encryptionkey@0"
I20260810 00:49:58.872201 22377 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 00:49:58.875558 22397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:49:58.876731 22377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 00:49:58.876906 22377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-0/wal
uuid: "8128a63efa534cf29e1d5e22750d38ea"
format_stamp: "Formatted at 2026-08-10 00:49:58 on dist-test-slave-1nl6"
server_key: "fa415bf75b5885bab9925d4645f50930"
server_key_iv: "2548b9a9f290ae227d66d60c7bec81d4"
server_key_version: "encryptionkey@0"
I20260810 00:49:58.877096 22377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:49:58.903340 22377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:49:58.907794 22377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:49:58.908105 22377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:49:58.909114 22377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:49:58.910466 22377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:49:58.910578 22377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:49:58.910733 22377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:49:58.910804 22377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:49:58.944834 22377 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.1:43213
I20260810 00:49:58.944923 22586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.1:43213 every 8 connection(s)
I20260810 00:49:58.946135 22377 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
I20260810 00:49:58.955988 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 22377
I20260810 00:49:58.956211 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance
I20260810 00:49:58.957054 28696 external_mini_cluster.cc:1428] Setting key d06b71dd7172af9093b8776c6fdf231a
Aug 10 00:49:58 dist-test-slave-1nl6 krb5kdc[22197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322998, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:49:58.966470 22592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:58.948763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:50151 (local address 127.28.6.62:39453)
0810 00:49:58.949267 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0810 00:49:58.949273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:49:58.950168 (+ 895us) server_negotiation.cc:407] Connection header received
0810 00:49:58.951965 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:49:58.951970 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:49:58.952053 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:49:58.952166 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:49:58.953648 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:58.954353 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:58.955855 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:58.956479 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:58.956633 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:58.960579 (+ 3946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:49:58.960584 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:49:58.960587 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:49:58.960619 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:49:58.963218 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:49:58.964158 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:49:58.964162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:49:58.964164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:49:58.964280 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:49:58.964777 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:49:58.964780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:49:58.964782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:49:58.965026 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:49:58.965135 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 00:49:58.966057 (+ 922us) server_negotiation.cc:299] Negotiation successful
0810 00:49:58.966221 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":237,"threads_started":1}
I20260810 00:49:58.968043 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:58.949264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39453 (local address 127.28.6.1:50151)
0810 00:49:58.950003 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:58.950042 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 00:49:58.951724 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:49:58.952433 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:49:58.952440 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:49:58.952944 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:49:58.953494 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:49:58.953512 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:49:58.954497 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:49:58.954500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:49:58.955691 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:49:58.955708 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:49:58.956697 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:49:58.956702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:49:58.956885 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:58.957597 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:49:58.957618 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:49:58.960338 (+ 2720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:49:58.963364 (+ 3026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:49:58.963375 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:49:58.963393 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:49:58.964035 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:49:58.964386 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:49:58.964393 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:49:58.964399 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:49:58.964653 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:49:58.965152 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:49:58.965163 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:49:58.965626 (+ 463us) client_negotiation.cc:769] Sending connection context
0810 00:49:58.965949 (+ 323us) client_negotiation.cc:240] Negotiation successful
0810 00:49:58.966522 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":277,"threads_started":1}
I20260810 00:49:58.969626 22587 heartbeater.cc:347] Connected to a master server at 127.28.6.62:39453
I20260810 00:49:58.969936 22587 heartbeater.cc:464] Registering TS with master...
I20260810 00:49:58.970646 22587 heartbeater.cc:511] Master 127.28.6.62:39453 requested a full tablet report, sending...
I20260810 00:49:58.972853 22271 ts_manager.cc:194] Registered new tserver with Master: 8128a63efa534cf29e1d5e22750d38ea (127.28.6.1:43213)
I20260810 00:49:58.974520 22271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.1@KRBTEST.COM'} at 127.28.6.1:50151
WARNING: no policy specified for kudu/127.28.6.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab.
Entry for principal kudu/127.28.6.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab.
I20260810 00:49:59.022284 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.2:0
--local_ip_for_outbound_sockets=127.28.6.2
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:39453
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:37241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:49:59.355224 22600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:49:59.355490 22600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:49:59.355669 22600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:49:59.362669 22600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:49:59.362773 22600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:49:59.362952 22600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.2
I20260810 00:49:59.372530 22600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:37241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:39453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.2
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:49:59.373818 22600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:49:59.375217 22600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:49:59.388495 22609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:49:59.388553 22614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:49:59.388549 22610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:49:59.389153 22600 server_base.cc:1034] running on GCE node
I20260810 00:49:59.389771 22600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:49:59.390650 22600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:49:59.391860 22600 hybrid_clock.cc:648] HybridClock initialized: now 1786322999391778 us; error 59 us; skew 500 ppm
I20260810 00:49:59.392203 22600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:49:59 dist-test-slave-1nl6 krb5kdc[22197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322999, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:49:59.395341 22600 init.cc:377] Logged in from keytab as kudu/127.28.6.2@KRBTEST.COM (short username kudu)
I20260810 00:49:59.396750 22600 webserver.cc:460] Webserver started at http://127.28.6.2:36419/ using document root <none> and password file <none>
I20260810 00:49:59.397431 22600 fs_manager.cc:359] Metadata directory not provided
I20260810 00:49:59.397583 22600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:49:59.397954 22600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:49:59.400998 22600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-1/data/instance:
uuid: "4247dc3fa8f34c7bb314fb78609dd96a"
format_stamp: "Formatted at 2026-08-10 00:49:59 on dist-test-slave-1nl6"
server_key: "11e259b13e22e44f0b9113883580ec3e"
server_key_iv: "993a55ac437ca5d235b4386730b7b18e"
server_key_version: "encryptionkey@0"
I20260810 00:49:59.401785 22600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance:
uuid: "4247dc3fa8f34c7bb314fb78609dd96a"
format_stamp: "Formatted at 2026-08-10 00:49:59 on dist-test-slave-1nl6"
server_key: "11e259b13e22e44f0b9113883580ec3e"
server_key_iv: "993a55ac437ca5d235b4386730b7b18e"
server_key_version: "encryptionkey@0"
I20260810 00:49:59.406817 22600 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260810 00:49:59.410391 22625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:49:59.411710 22600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 00:49:59.411903 22600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-1/wal
uuid: "4247dc3fa8f34c7bb314fb78609dd96a"
format_stamp: "Formatted at 2026-08-10 00:49:59 on dist-test-slave-1nl6"
server_key: "11e259b13e22e44f0b9113883580ec3e"
server_key_iv: "993a55ac437ca5d235b4386730b7b18e"
server_key_version: "encryptionkey@0"
I20260810 00:49:59.412104 22600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:49:59.435669 22600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:49:59.439641 22600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:49:59.439946 22600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:49:59.440982 22600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:49:59.442247 22600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:49:59.442351 22600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:49:59.442499 22600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:49:59.442574 22600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:49:59.478011 22600 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.2:38865
I20260810 00:49:59.478071 22815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.2:38865 every 8 connection(s)
I20260810 00:49:59.479373 22600 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
I20260810 00:49:59.479908 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 22600
I20260810 00:49:59.480103 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance
I20260810 00:49:59.480743 28696 external_mini_cluster.cc:1428] Setting key 3bc8739b1408ce6521bb39a21faac614
Aug 10 00:49:59 dist-test-slave-1nl6 krb5kdc[22197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322999, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:49:59.501258 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:59.482469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:34815 (local address 127.28.6.62:39453)
0810 00:49:59.483335 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0810 00:49:59.483340 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:49:59.484088 (+ 748us) server_negotiation.cc:407] Connection header received
0810 00:49:59.486528 (+ 2440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:49:59.486535 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:49:59.486620 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:49:59.486750 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:49:59.488297 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:59.489027 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:59.490538 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:49:59.491064 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:49:59.491199 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:59.495274 (+ 4075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:49:59.495283 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:49:59.495290 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:49:59.495332 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:49:59.498095 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:49:59.498811 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:49:59.498825 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:49:59.498828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:49:59.499033 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:49:59.499486 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:49:59.499491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:49:59.499495 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:49:59.499777 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:49:59.499970 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0810 00:49:59.500627 (+ 657us) server_negotiation.cc:299] Negotiation successful
0810 00:49:59.500859 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":649,"threads_started":1}
I20260810 00:49:59.502147 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:49:59.483120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39453 (local address 127.28.6.2:34815)
0810 00:49:59.483938 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0810 00:49:59.483975 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 00:49:59.486266 (+ 2291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:49:59.487026 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:49:59.487034 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:49:59.487547 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:49:59.488119 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:49:59.488135 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:49:59.489183 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:49:59.489195 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:49:59.490387 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:49:59.490400 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:49:59.491307 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:49:59.491315 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:49:59.491539 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:49:59.492251 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:49:59.492273 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:49:59.495033 (+ 2760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:49:59.498226 (+ 3193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:49:59.498233 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:49:59.498244 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:49:59.498664 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:49:59.499153 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:49:59.499158 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:49:59.499164 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:49:59.499355 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:49:59.499924 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:49:59.499934 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:49:59.500254 (+ 320us) client_negotiation.cc:769] Sending connection context
0810 00:49:59.500519 (+ 265us) client_negotiation.cc:240] Negotiation successful
0810 00:49:59.501037 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":352,"threads_started":1}
I20260810 00:49:59.503638 22816 heartbeater.cc:347] Connected to a master server at 127.28.6.62:39453
I20260810 00:49:59.503968 22816 heartbeater.cc:464] Registering TS with master...
I20260810 00:49:59.504685 22816 heartbeater.cc:511] Master 127.28.6.62:39453 requested a full tablet report, sending...
I20260810 00:49:59.506178 22271 ts_manager.cc:194] Registered new tserver with Master: 4247dc3fa8f34c7bb314fb78609dd96a (127.28.6.2:38865)
I20260810 00:49:59.507081 22271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.2@KRBTEST.COM'} at 127.28.6.2:34815
WARNING: no policy specified for kudu/127.28.6.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab.
Entry for principal kudu/127.28.6.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab.
I20260810 00:49:59.550251 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.3:0
--local_ip_for_outbound_sockets=127.28.6.3
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:39453
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:37241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:49:59.913066 22824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:49:59.913491 22824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:49:59.913673 22824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:49:59.924597 22824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:49:59.924701 22824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:49:59.924815 22824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.3
I20260810 00:49:59.934748 22824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:37241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:39453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.3
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:49:59.936015 22824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:49:59.937351 22824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:49:59.950992 22833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:49:59.950951 22838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:49:59.951040 22840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:49:59.952014 22824 server_base.cc:1034] running on GCE node
I20260810 00:49:59.952618 22824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:49:59.953507 22824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:49:59.954741 22824 hybrid_clock.cc:648] HybridClock initialized: now 1786322999954685 us; error 67 us; skew 500 ppm
I20260810 00:49:59.955122 22824 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:49:59 dist-test-slave-1nl6 krb5kdc[22197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322999, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:49:59.958204 22824 init.cc:377] Logged in from keytab as kudu/127.28.6.3@KRBTEST.COM (short username kudu)
I20260810 00:49:59.959587 22824 webserver.cc:460] Webserver started at http://127.28.6.3:33657/ using document root <none> and password file <none>
I20260810 00:49:59.960222 22824 fs_manager.cc:359] Metadata directory not provided
I20260810 00:49:59.960371 22824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:49:59.960664 22824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:49:59.963516 22824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-2/data/instance:
uuid: "3a6614ae032d42329a7cbb5d28490d05"
format_stamp: "Formatted at 2026-08-10 00:49:59 on dist-test-slave-1nl6"
server_key: "162bb8a9a015b057c3b707fa5d7e13c8"
server_key_iv: "751c4e8dd050af47d3f7c5e6e4af2844"
server_key_version: "encryptionkey@0"
I20260810 00:49:59.964246 22824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance:
uuid: "3a6614ae032d42329a7cbb5d28490d05"
format_stamp: "Formatted at 2026-08-10 00:49:59 on dist-test-slave-1nl6"
server_key: "162bb8a9a015b057c3b707fa5d7e13c8"
server_key_iv: "751c4e8dd050af47d3f7c5e6e4af2844"
server_key_version: "encryptionkey@0"
I20260810 00:49:59.969183 22824 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 00:49:59.972550 22853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:49:59.973766 22824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 00:49:59.973912 22824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-2/wal
uuid: "3a6614ae032d42329a7cbb5d28490d05"
format_stamp: "Formatted at 2026-08-10 00:49:59 on dist-test-slave-1nl6"
server_key: "162bb8a9a015b057c3b707fa5d7e13c8"
server_key_iv: "751c4e8dd050af47d3f7c5e6e4af2844"
server_key_version: "encryptionkey@0"
I20260810 00:49:59.974079 22824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:49:59.978192 22587 heartbeater.cc:503] Master 127.28.6.62:39453 was elected leader, sending a full tablet report...
I20260810 00:50:00.001348 22824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:50:00.005582 22824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:50:00.005900 22824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:50:00.006968 22824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:50:00.008350 22824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:50:00.008458 22824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:00.008608 22824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:50:00.008682 22824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:00.043167 22824 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.3:45715
I20260810 00:50:00.043269 23050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.3:45715 every 8 connection(s)
I20260810 00:50:00.044476 22824 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
I20260810 00:50:00.052033 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 22824
I20260810 00:50:00.052233 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance
I20260810 00:50:00.052723 28696 external_mini_cluster.cc:1428] Setting key 3c0192838a3f9a7de99d2dd0775439e2
Aug 10 00:50:00 dist-test-slave-1nl6 krb5kdc[22197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322999, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:50:00.065280 23056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:00.047178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:39773 (local address 127.28.6.62:39453)
0810 00:50:00.047676 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:00.047681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:00.048720 (+ 1039us) server_negotiation.cc:407] Connection header received
0810 00:50:00.050714 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:00.050720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:00.050817 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:00.050965 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:00.052717 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:00.053829 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:00.055926 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:00.056428 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:00.056536 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:00.059964 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:00.059969 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:00.059973 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:50:00.060006 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:00.062486 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:00.063279 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:00.063283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:00.063285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:00.063402 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:00.063846 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:00.063849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:00.063851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:00.064099 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:00.064211 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:00.064852 (+ 641us) server_negotiation.cc:299] Negotiation successful
0810 00:50:00.065020 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":257,"threads_started":1}
I20260810 00:50:00.066215 23057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:00.047779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39453 (local address 127.28.6.3:39773)
0810 00:50:00.048550 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:00.048598 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:00.050418 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:00.051219 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:00.051227 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:00.051810 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:50:00.052531 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:00.052551 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:00.054020 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:00.054028 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:00.055765 (+ 1737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:00.055778 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:00.056627 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:00.056632 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:00.056820 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:00.057464 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:50:00.057485 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:50:00.059770 (+ 2285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:50:00.062615 (+ 2845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:50:00.062622 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:50:00.062633 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:50:00.063169 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:50:00.063545 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:50:00.063548 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:50:00.063551 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:50:00.063736 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:50:00.064210 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:50:00.064216 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:50:00.064523 (+ 307us) client_negotiation.cc:769] Sending connection context
0810 00:50:00.064745 (+ 222us) client_negotiation.cc:240] Negotiation successful
0810 00:50:00.065106 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":309,"threads_started":1}
I20260810 00:50:00.067663 23053 heartbeater.cc:347] Connected to a master server at 127.28.6.62:39453
I20260810 00:50:00.068012 23053 heartbeater.cc:464] Registering TS with master...
I20260810 00:50:00.068751 23053 heartbeater.cc:511] Master 127.28.6.62:39453 requested a full tablet report, sending...
I20260810 00:50:00.070248 22270 ts_manager.cc:194] Registered new tserver with Master: 3a6614ae032d42329a7cbb5d28490d05 (127.28.6.3:45715)
I20260810 00:50:00.071256 22270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.3@KRBTEST.COM'} at 127.28.6.3:39773
I20260810 00:50:00.080562 28696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 00:50:00.098136 23056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:00.085408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40938 (local address 127.28.6.62:39453)
0810 00:50:00.085620 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:00.085627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:00.086022 (+ 395us) server_negotiation.cc:407] Connection header received
0810 00:50:00.086151 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:00.086155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:00.086215 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:00.086291 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:00.087892 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:00.088599 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:00.089967 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:00.090340 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:00.090429 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:00.093000 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:00.093005 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:00.093008 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:50:00.093037 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:00.095723 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:00.096338 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:00.096343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:00.096347 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:00.096461 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:00.096800 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:00.096803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:00.096805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:00.097112 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:00.097288 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:00.097592 (+ 304us) server_negotiation.cc:299] Negotiation successful
0810 00:50:00.097759 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 00:50:00.102145 22270 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40938:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 00:50:00.109539 22270 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 00:50:00.162174 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:00.122200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48294 (local address 127.28.6.1:43213)
0810 00:50:00.152999 (+ 30799us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:00.153009 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:00.153043 (+ 34us) server_negotiation.cc:407] Connection header received
0810 00:50:00.153154 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:00.153162 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:00.153478 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:00.153702 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:50:00.154882 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:00.156501 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:00.159281 (+ 2780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:00.160899 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:00.161056 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:00.161317 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:00.161619 (+ 302us) server_negotiation.cc:299] Negotiation successful
0810 00:50:00.161802 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30620,"thread_start_us":30400,"threads_started":1}
I20260810 00:50:00.162293 23078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:00.121887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:43213 (local address 127.0.0.1:48294)
0810 00:50:00.134234 (+ 12347us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:00.134286 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:00.134498 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:00.154030 (+ 19532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:00.154050 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:00.154092 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:50:00.154662 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:00.154677 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:00.156681 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:00.156685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:00.159108 (+ 2423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:00.159124 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:00.161140 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:00.161147 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:00.161379 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:00.161419 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 00:50:00.161560 (+ 141us) client_negotiation.cc:240] Negotiation successful
0810 00:50:00.161755 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12028,"mutex_wait_us":26,"thread_start_us":32264,"threads_started":1}
I20260810 00:50:00.162941 23085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:00.121891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:45715 (local address 127.0.0.1:43440)
0810 00:50:00.152068 (+ 30177us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:00.152094 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:00.152274 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:00.155420 (+ 3146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:00.155425 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:00.155455 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:50:00.155801 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:00.155814 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:00.158235 (+ 2421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:00.158242 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:00.160207 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:00.160220 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:00.162016 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:00.162022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:00.162207 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:00.162256 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 00:50:00.162366 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 00:50:00.162510 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30004,"mutex_wait_us":103,"thread_start_us":12775,"threads_started":1}
I20260810 00:50:00.163406 23084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:00.122327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43440 (local address 127.28.6.3:45715)
0810 00:50:00.154751 (+ 32424us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:00.154763 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:00.154802 (+ 39us) server_negotiation.cc:407] Connection header received
0810 00:50:00.154945 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:00.154953 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:00.155198 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:00.155418 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:50:00.156050 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:00.158101 (+ 2051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:00.160378 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:00.161524 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:00.161647 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:00.161858 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:00.162833 (+ 975us) server_negotiation.cc:299] Negotiation successful
0810 00:50:00.163038 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32212,"thread_start_us":31995,"threads_started":1}
I20260810 00:50:00.164964 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:00.122878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:38865 (local address 127.0.0.1:49448)
0810 00:50:00.155163 (+ 32285us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:00.155183 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:00.155326 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:00.156064 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:00.156068 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:00.156121 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:50:00.156491 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:00.156502 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:00.158466 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:00.158472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:00.160897 (+ 2425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:00.160912 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:00.164055 (+ 3143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:00.164065 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:00.164256 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:00.164291 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 00:50:00.164399 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 00:50:00.164525 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32138,"mutex_wait_us":1,"thread_start_us":28794,"threads_started":1}
I20260810 00:50:00.165886 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:00.123617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49448 (local address 127.28.6.2:38865)
0810 00:50:00.151468 (+ 27851us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:00.151483 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:00.155311 (+ 3828us) server_negotiation.cc:407] Connection header received
0810 00:50:00.155555 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:00.155564 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:00.155793 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:00.155920 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:50:00.156658 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:00.158306 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:00.161067 (+ 2761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:00.163732 (+ 2665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:00.163882 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:00.164073 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:00.165354 (+ 1281us) server_negotiation.cc:299] Negotiation successful
0810 00:50:00.165534 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27652,"thread_start_us":27506,"threads_started":1}
I20260810 00:50:00.166616 22928 tablet_service.cc:1467] Processing CreateTablet for tablet b55a0fc859004296b58a6f23c9454cbf (DEFAULT_TABLE table=test-table [id=c4297862bb0e431c9916268c65e996e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:50:00.166779 22477 tablet_service.cc:1467] Processing CreateTablet for tablet b55a0fc859004296b58a6f23c9454cbf (DEFAULT_TABLE table=test-table [id=c4297862bb0e431c9916268c65e996e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:50:00.168270 22477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b55a0fc859004296b58a6f23c9454cbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:50:00.168785 22928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b55a0fc859004296b58a6f23c9454cbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:50:00.169423 22714 tablet_service.cc:1467] Processing CreateTablet for tablet b55a0fc859004296b58a6f23c9454cbf (DEFAULT_TABLE table=test-table [id=c4297862bb0e431c9916268c65e996e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:50:00.170704 22714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b55a0fc859004296b58a6f23c9454cbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:50:00.180766 23090 tablet_bootstrap.cc:492] T b55a0fc859004296b58a6f23c9454cbf P 8128a63efa534cf29e1d5e22750d38ea: Bootstrap starting.
I20260810 00:50:00.184971 23090 tablet_bootstrap.cc:654] T b55a0fc859004296b58a6f23c9454cbf P 8128a63efa534cf29e1d5e22750d38ea: Neither blocks nor log segments found. Creating new log.
I20260810 00:50:00.186071 23090 log.cc:824] T b55a0fc859004296b58a6f23c9454cbf P 8128a63efa534cf29e1d5e22750d38ea: Log is configured to *not* fsync() on all Append() calls
I20260810 00:50:00.187871 23091 tablet_bootstrap.cc:492] T b55a0fc859004296b58a6f23c9454cbf P 3a6614ae032d42329a7cbb5d28490d05: Bootstrap starting.
I20260810 00:50:00.191237 23092 tablet_bootstrap.cc:492] T b55a0fc859004296b58a6f23c9454cbf P 4247dc3fa8f34c7bb314fb78609dd96a: Bootstrap starting.
I20260810 00:50:00.192710 23091 tablet_bootstrap.cc:654] T b55a0fc859004296b58a6f23c9454cbf P 3a6614ae032d42329a7cbb5d28490d05: Neither blocks nor log segments found. Creating new log.
I20260810 00:50:00.194085 23091 log.cc:824] T b55a0fc859004296b58a6f23c9454cbf P 3a6614ae032d42329a7cbb5d28490d05: Log is configured to *not* fsync() on all Append() calls
I20260810 00:50:00.194571 23092 tablet_bootstrap.cc:654] T b55a0fc859004296b58a6f23c9454cbf P 4247dc3fa8f34c7bb314fb78609dd96a: Neither blocks nor log segments found. Creating new log.
I20260810 00:50:00.195163 23090 tablet_bootstrap.cc:492] T b55a0fc859004296b58a6f23c9454cbf P 8128a63efa534cf29e1d5e22750d38ea: No bootstrap required, opened a new log
I20260810 00:50:00.195585 23090 ts_tablet_manager.cc:1397] T b55a0fc859004296b58a6f23c9454cbf P 8128a63efa534cf29e1d5e22750d38ea: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.008s
I20260810 00:50:00.195863 23092 log.cc:824] T b55a0fc859004296b58a6f23c9454cbf P 4247dc3fa8f34c7bb314fb78609dd96a: Log is configured to *not* fsync() on all Append() calls
I20260810 00:50:00.198431 23092 tablet_bootstrap.cc:492] T b55a0fc859004296b58a6f23c9454cbf P 4247dc3fa8f34c7bb314fb78609dd96a: No bootstrap required, opened a new log
I20260810 00:50:00.198694 23092 ts_tablet_manager.cc:1397] T b55a0fc859004296b58a6f23c9454cbf P 4247dc3fa8f34c7bb314fb78609dd96a: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260810 00:50:00.203728 23090 raft_consensus.cc:348] T b55a0fc859004296b58a6f23c9454cbf P 8128a63efa534cf29e1d5e22750d38ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8128a63efa534cf29e1d5e22750d38ea" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 43213 } } peers { permanent_uuid: "3a6614ae032d42329a7cbb5d28490d05" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45715 } } peers { permanent_uuid: "4247dc3fa8f34c7bb314fb78609dd96a" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 38865 } }
I20260810 00:50:00.204087 23090 raft_consensus.cc:374] T b55a0fc859004296b58a6f23c9454cbf P 8128a63efa534cf29e1d5e22750d38ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:50:00.204253 23090 raft_consensus.cc:729] T b55a0fc859004296b58a6f23c9454cbf P 8128a63efa534cf29e1d5e22750d38ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8128a63efa534cf29e1d5e22750d38ea, State: Initialized, Role: FOLLOWER
I20260810 00:50:00.204890 23090 consensus_queue.cc:260] T b55a0fc859004296b58a6f23c9454cbf P 8128a63efa534cf29e1d5e22750d38ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8128a63efa534cf29e1d5e22750d38ea" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 43213 } } peers { permanent_uuid: "3a6614ae032d42329a7cbb5d28490d05" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45715 } } peers { permanent_uuid: "4247dc3fa8f34c7bb314fb78609dd96a" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 38865 } }
I20260810 00:50:00.206116 23091 tablet_bootstrap.cc:492] T b55a0fc859004296b58a6f23c9454cbf P 3a6614ae032d42329a7cbb5d28490d05: No bootstrap required, opened a new log
I20260810 00:50:00.206434 23091 ts_tablet_manager.cc:1397] T b55a0fc859004296b58a6f23c9454cbf P 3a6614ae032d42329a7cbb5d28490d05: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.004s
I20260810 00:50:00.206184 23092 raft_consensus.cc:348] T b55a0fc859004296b58a6f23c9454cbf P 4247dc3fa8f34c7bb314fb78609dd96a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8128a63efa534cf29e1d5e22750d38ea" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 43213 } } peers { permanent_uuid: "3a6614ae032d42329a7cbb5d28490d05" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45715 } } peers { permanent_uuid: "4247dc3fa8f34c7bb314fb78609dd96a" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 38865 } }
I20260810 00:50:00.206633 23092 raft_consensus.cc:374] T b55a0fc859004296b58a6f23c9454cbf P 4247dc3fa8f34c7bb314fb78609dd96a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:50:00.206776 23090 ts_tablet_manager.cc:1428] T b55a0fc859004296b58a6f23c9454cbf P 8128a63efa534cf29e1d5e22750d38ea: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260810 00:50:00.206770 23092 raft_consensus.cc:729] T b55a0fc859004296b58a6f23c9454cbf P 4247dc3fa8f34c7bb314fb78609dd96a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4247dc3fa8f34c7bb314fb78609dd96a, State: Initialized, Role: FOLLOWER
I20260810 00:50:00.207508 23092 consensus_queue.cc:260] T b55a0fc859004296b58a6f23c9454cbf P 4247dc3fa8f34c7bb314fb78609dd96a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8128a63efa534cf29e1d5e22750d38ea" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 43213 } } peers { permanent_uuid: "3a6614ae032d42329a7cbb5d28490d05" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45715 } } peers { permanent_uuid: "4247dc3fa8f34c7bb314fb78609dd96a" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 38865 } }
I20260810 00:50:00.209159 22816 heartbeater.cc:503] Master 127.28.6.62:39453 was elected leader, sending a full tablet report...
I20260810 00:50:00.209753 23092 ts_tablet_manager.cc:1428] T b55a0fc859004296b58a6f23c9454cbf P 4247dc3fa8f34c7bb314fb78609dd96a: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260810 00:50:00.216001 23091 raft_consensus.cc:348] T b55a0fc859004296b58a6f23c9454cbf P 3a6614ae032d42329a7cbb5d28490d05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8128a63efa534cf29e1d5e22750d38ea" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 43213 } } peers { permanent_uuid: "3a6614ae032d42329a7cbb5d28490d05" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45715 } } peers { permanent_uuid: "4247dc3fa8f34c7bb314fb78609dd96a" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 38865 } }
I20260810 00:50:00.216364 23091 raft_consensus.cc:374] T b55a0fc859004296b58a6f23c9454cbf P 3a6614ae032d42329a7cbb5d28490d05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:50:00.216547 23091 raft_consensus.cc:729] T b55a0fc859004296b58a6f23c9454cbf P 3a6614ae032d42329a7cbb5d28490d05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a6614ae032d42329a7cbb5d28490d05, State: Initialized, Role: FOLLOWER
I20260810 00:50:00.217089 23091 consensus_queue.cc:260] T b55a0fc859004296b58a6f23c9454cbf P 3a6614ae032d42329a7cbb5d28490d05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8128a63efa534cf29e1d5e22750d38ea" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 43213 } } peers { permanent_uuid: "3a6614ae032d42329a7cbb5d28490d05" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45715 } } peers { permanent_uuid: "4247dc3fa8f34c7bb314fb78609dd96a" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 38865 } }
I20260810 00:50:00.218081 23053 heartbeater.cc:503] Master 127.28.6.62:39453 was elected leader, sending a full tablet report...
I20260810 00:50:00.218468 23091 ts_tablet_manager.cc:1428] T b55a0fc859004296b58a6f23c9454cbf P 3a6614ae032d42329a7cbb5d28490d05: Time spent starting tablet: real 0.012s user 0.010s sys 0.001s
W20260810 00:50:00.231559 22817 tablet.cc:2369] T b55a0fc859004296b58a6f23c9454cbf P 4247dc3fa8f34c7bb314fb78609dd96a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 00:50:00.296289 23054 tablet.cc:2369] T b55a0fc859004296b58a6f23c9454cbf P 3a6614ae032d42329a7cbb5d28490d05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:50:00.354470 23101 raft_consensus.cc:482] T b55a0fc859004296b58a6f23c9454cbf P 8128a63efa534cf29e1d5e22750d38ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:50:00.354812 23101 raft_consensus.cc:504] T b55a0fc859004296b58a6f23c9454cbf P 8128a63efa534cf29e1d5e22750d38ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8128a63efa534cf29e1d5e22750d38ea" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 43213 } } peers { permanent_uuid: "3a6614ae032d42329a7cbb5d28490d05" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45715 } } peers { permanent_uuid: "4247dc3fa8f34c7bb314fb78609dd96a" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 38865 } }
I20260810 00:50:00.356609 23101 leader_election.cc:288] T b55a0fc859004296b58a6f23c9454cbf P 8128a63efa534cf29e1d5e22750d38ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a6614ae032d42329a7cbb5d28490d05 (127.28.6.3:45715), 4247dc3fa8f34c7bb314fb78609dd96a (127.28.6.2:38865)
I20260810 00:50:00.365768 23084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:00.357154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:47187 (local address 127.28.6.3:45715)
0810 00:50:00.357404 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:00.357412 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:00.357865 (+ 453us) server_negotiation.cc:407] Connection header received
0810 00:50:00.358125 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:00.358131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:00.358212 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:00.358319 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:50:00.359176 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:00.360311 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:00.363308 (+ 2997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:00.364555 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:00.364684 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:00.364809 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:00.365358 (+ 549us) server_negotiation.cc:299] Negotiation successful
0810 00:50:00.365488 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 00:50:00.365764 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:00.357088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:51945 (local address 127.28.6.2:38865)
0810 00:50:00.357339 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:00.357349 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:00.357983 (+ 634us) server_negotiation.cc:407] Connection header received
0810 00:50:00.358144 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:00.358149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:00.358218 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:00.358319 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:50:00.359176 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:00.360311 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:00.363309 (+ 2998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:00.364553 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:00.364684 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:00.364782 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:00.365358 (+ 576us) server_negotiation.cc:299] Negotiation successful
0810 00:50:00.365488 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 00:50:00.366115 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:00.357037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:38865 (local address 127.28.6.1:51945)
0810 00:50:00.357881 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:00.357905 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:00.358017 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:00.358355 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:00.358359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:00.358453 (+ 94us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:50:00.358979 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:00.359002 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:00.360453 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:00.360457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:00.363137 (+ 2680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:00.363153 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:00.364827 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:00.364836 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:00.365085 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:00.365212 (+ 127us) client_negotiation.cc:769] Sending connection context
0810 00:50:00.365310 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 00:50:00.365541 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"mutex_wait_us":297,"spinlock_wait_cycles":26496,"thread_start_us":539,"threads_started":1}
I20260810 00:50:00.366137 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:00.356882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:45715 (local address 127.28.6.1:47187)
0810 00:50:00.357725 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:00.357751 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:00.357933 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:00.358352 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:00.358357 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:00.358386 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:50:00.359024 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:00.359040 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:00.360451 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:00.360456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:00.363137 (+ 2681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:00.363153 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:00.364850 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:00.364855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:00.365086 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:00.365209 (+ 123us) client_negotiation.cc:769] Sending connection context
0810 00:50:00.365310 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 00:50:00.365541 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"mutex_wait_us":348,"thread_start_us":469,"threads_started":1}
I20260810 00:50:00.367373 22969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b55a0fc859004296b58a6f23c9454cbf" candidate_uuid: "8128a63efa534cf29e1d5e22750d38ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a6614ae032d42329a7cbb5d28490d05" is_pre_election: true
I20260810 00:50:00.367373 22746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b55a0fc859004296b58a6f23c9454cbf" candidate_uuid: "8128a63efa534cf29e1d5e22750d38ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4247dc3fa8f34c7bb314fb78609dd96a" is_pre_election: true
I20260810 00:50:00.367977 22746 raft_consensus.cc:2454] T b55a0fc859004296b58a6f23c9454cbf P 4247dc3fa8f34c7bb314fb78609dd96a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8128a63efa534cf29e1d5e22750d38ea in term 0.
I20260810 00:50:00.367976 22969 raft_consensus.cc:2454] T b55a0fc859004296b58a6f23c9454cbf P 3a6614ae032d42329a7cbb5d28490d05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8128a63efa534cf29e1d5e22750d38ea in term 0.
I20260810 00:50:00.368655 22404 leader_election.cc:302] T b55a0fc859004296b58a6f23c9454cbf P 8128a63efa534cf29e1d5e22750d38ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a6614ae032d42329a7cbb5d28490d05, 8128a63efa534cf29e1d5e22750d38ea; no voters:
I20260810 00:50:00.368996 23101 raft_consensus.cc:2781] T b55a0fc859004296b58a6f23c9454cbf P 8128a63efa534cf29e1d5e22750d38ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 00:50:00.369161 23101 raft_consensus.cc:482] T b55a0fc859004296b58a6f23c9454cbf P 8128a63efa534cf29e1d5e22750d38ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 00:50:00.369287 23101 raft_consensus.cc:3037] T b55a0fc859004296b58a6f23c9454cbf P 8128a63efa534cf29e1d5e22750d38ea [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:50:00.371654 23101 raft_consensus.cc:504] T b55a0fc859004296b58a6f23c9454cbf P 8128a63efa534cf29e1d5e22750d38ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8128a63efa534cf29e1d5e22750d38ea" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 43213 } } peers { permanent_uuid: "3a6614ae032d42329a7cbb5d28490d05" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45715 } } peers { permanent_uuid: "4247dc3fa8f34c7bb314fb78609dd96a" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 38865 } }
I20260810 00:50:00.372313 23101 leader_election.cc:288] T b55a0fc859004296b58a6f23c9454cbf P 8128a63efa534cf29e1d5e22750d38ea [CANDIDATE]: Term 1 election: Requested vote from peers 3a6614ae032d42329a7cbb5d28490d05 (127.28.6.3:45715), 4247dc3fa8f34c7bb314fb78609dd96a (127.28.6.2:38865)
I20260810 00:50:00.372781 22969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b55a0fc859004296b58a6f23c9454cbf" candidate_uuid: "8128a63efa534cf29e1d5e22750d38ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a6614ae032d42329a7cbb5d28490d05"
I20260810 00:50:00.372781 22746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b55a0fc859004296b58a6f23c9454cbf" candidate_uuid: "8128a63efa534cf29e1d5e22750d38ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4247dc3fa8f34c7bb314fb78609dd96a"
I20260810 00:50:00.373003 22746 raft_consensus.cc:3037] T b55a0fc859004296b58a6f23c9454cbf P 4247dc3fa8f34c7bb314fb78609dd96a [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:50:00.372997 22969 raft_consensus.cc:3037] T b55a0fc859004296b58a6f23c9454cbf P 3a6614ae032d42329a7cbb5d28490d05 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:50:00.376260 22969 raft_consensus.cc:2454] T b55a0fc859004296b58a6f23c9454cbf P 3a6614ae032d42329a7cbb5d28490d05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8128a63efa534cf29e1d5e22750d38ea in term 1.
I20260810 00:50:00.376252 22746 raft_consensus.cc:2454] T b55a0fc859004296b58a6f23c9454cbf P 4247dc3fa8f34c7bb314fb78609dd96a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8128a63efa534cf29e1d5e22750d38ea in term 1.
I20260810 00:50:00.376700 22403 leader_election.cc:302] T b55a0fc859004296b58a6f23c9454cbf P 8128a63efa534cf29e1d5e22750d38ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4247dc3fa8f34c7bb314fb78609dd96a, 8128a63efa534cf29e1d5e22750d38ea; no voters:
I20260810 00:50:00.377044 23101 raft_consensus.cc:2781] T b55a0fc859004296b58a6f23c9454cbf P 8128a63efa534cf29e1d5e22750d38ea [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:50:00.377512 23101 raft_consensus.cc:686] T b55a0fc859004296b58a6f23c9454cbf P 8128a63efa534cf29e1d5e22750d38ea [term 1 LEADER]: Becoming Leader. State: Replica: 8128a63efa534cf29e1d5e22750d38ea, State: Running, Role: LEADER
I20260810 00:50:00.378048 23101 consensus_queue.cc:237] T b55a0fc859004296b58a6f23c9454cbf P 8128a63efa534cf29e1d5e22750d38ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8128a63efa534cf29e1d5e22750d38ea" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 43213 } } peers { permanent_uuid: "3a6614ae032d42329a7cbb5d28490d05" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45715 } } peers { permanent_uuid: "4247dc3fa8f34c7bb314fb78609dd96a" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 38865 } }
I20260810 00:50:00.382525 22270 catalog_manager.cc:5495] T b55a0fc859004296b58a6f23c9454cbf P 8128a63efa534cf29e1d5e22750d38ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 8128a63efa534cf29e1d5e22750d38ea (127.28.6.1). New cstate: current_term: 1 leader_uuid: "8128a63efa534cf29e1d5e22750d38ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8128a63efa534cf29e1d5e22750d38ea" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 43213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a6614ae032d42329a7cbb5d28490d05" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4247dc3fa8f34c7bb314fb78609dd96a" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 38865 } health_report { overall_health: UNKNOWN } } }
W20260810 00:50:00.448715 22588 tablet.cc:2369] T b55a0fc859004296b58a6f23c9454cbf P 8128a63efa534cf29e1d5e22750d38ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:50:00.449615 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:00.443989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48302 (local address 127.28.6.1:43213)
0810 00:50:00.444223 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:00.444230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:00.444250 (+ 20us) server_negotiation.cc:407] Connection header received
0810 00:50:00.444444 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:00.444451 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:00.444546 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:00.444699 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:50:00.445106 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:00.445835 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:00.447373 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:00.447798 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:00.447949 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:00.448446 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:50:00.449020 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:50:00.449171 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:00.449282 (+ 111us) server_negotiation.cc:299] Negotiation successful
0810 00:50:00.449359 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 00:50:00.457594 22746 raft_consensus.cc:1261] T b55a0fc859004296b58a6f23c9454cbf P 4247dc3fa8f34c7bb314fb78609dd96a [term 1 FOLLOWER]: Refusing update from remote peer 8128a63efa534cf29e1d5e22750d38ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:50:00.457649 22969 raft_consensus.cc:1261] T b55a0fc859004296b58a6f23c9454cbf P 3a6614ae032d42329a7cbb5d28490d05 [term 1 FOLLOWER]: Refusing update from remote peer 8128a63efa534cf29e1d5e22750d38ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:50:00.458590 23111 consensus_queue.cc:1035] T b55a0fc859004296b58a6f23c9454cbf P 8128a63efa534cf29e1d5e22750d38ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a6614ae032d42329a7cbb5d28490d05" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:50:00.458994 23101 consensus_queue.cc:1035] T b55a0fc859004296b58a6f23c9454cbf P 8128a63efa534cf29e1d5e22750d38ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "4247dc3fa8f34c7bb314fb78609dd96a" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:50:00.472929 23123 mvcc.cc:205] Tried to move back new op lower bound from 7316779009864794112 to 7316779009552306176. Current Snapshot: MvccSnapshot[applied={T|T < 7316779009864794112}]
I20260810 00:50:00.488547 22270 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:50:00.488816 22270 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:50:00.492719 22270 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 21c24236ed894a9495841b718ebda2c5: Sending DeleteTablet for 3 replicas of tablet b55a0fc859004296b58a6f23c9454cbf
I20260810 00:50:00.493829 22477 tablet_service.cc:1514] Processing DeleteTablet for tablet b55a0fc859004296b58a6f23c9454cbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:50:00 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:48294
I20260810 00:50:00.493965 22714 tablet_service.cc:1514] Processing DeleteTablet for tablet b55a0fc859004296b58a6f23c9454cbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:50:00 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:49448
I20260810 00:50:00.495093 22928 tablet_service.cc:1514] Processing DeleteTablet for tablet b55a0fc859004296b58a6f23c9454cbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:50:00 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:43440
I20260810 00:50:00.495085 23119 mvcc.cc:205] Tried to move back new op lower bound from 7316779009864794112 to 7316779009552306176. Current Snapshot: MvccSnapshot[applied={T|T < 7316779009864794112}]
I20260810 00:50:00.496114 23138 tablet_replica.cc:330] T b55a0fc859004296b58a6f23c9454cbf P 3a6614ae032d42329a7cbb5d28490d05: stopping tablet replica
I20260810 00:50:00.497537 23138 raft_consensus.cc:2229] T b55a0fc859004296b58a6f23c9454cbf P 3a6614ae032d42329a7cbb5d28490d05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:50:00.497879 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 22377
I20260810 00:50:00.498539 23138 raft_consensus.cc:2258] T b55a0fc859004296b58a6f23c9454cbf P 3a6614ae032d42329a7cbb5d28490d05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:50:00.505707 23138 ts_tablet_manager.cc:1905] T b55a0fc859004296b58a6f23c9454cbf P 3a6614ae032d42329a7cbb5d28490d05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:50:00.512612 23138 ts_tablet_manager.cc:1918] T b55a0fc859004296b58a6f23c9454cbf P 3a6614ae032d42329a7cbb5d28490d05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:50:00.512830 23138 log.cc:1196] T b55a0fc859004296b58a6f23c9454cbf P 3a6614ae032d42329a7cbb5d28490d05: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-2/wal/wals/b55a0fc859004296b58a6f23c9454cbf
I20260810 00:50:00.513324 23138 ts_tablet_manager.cc:1939] T b55a0fc859004296b58a6f23c9454cbf P 3a6614ae032d42329a7cbb5d28490d05: Deleting consensus metadata
I20260810 00:50:00.514760 22249 catalog_manager.cc:4841] TS 3a6614ae032d42329a7cbb5d28490d05 (127.28.6.3:45715): tablet b55a0fc859004296b58a6f23c9454cbf (table test-table [id=c4297862bb0e431c9916268c65e996e4]) successfully deleted
I20260810 00:50:00.520131 23137 tablet_replica.cc:330] T b55a0fc859004296b58a6f23c9454cbf P 4247dc3fa8f34c7bb314fb78609dd96a: stopping tablet replica
I20260810 00:50:00.520538 23137 raft_consensus.cc:2229] T b55a0fc859004296b58a6f23c9454cbf P 4247dc3fa8f34c7bb314fb78609dd96a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:50:00.521083 23137 raft_consensus.cc:2258] T b55a0fc859004296b58a6f23c9454cbf P 4247dc3fa8f34c7bb314fb78609dd96a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:50:00.521844 23136 tablet_replica.cc:330] T b55a0fc859004296b58a6f23c9454cbf P 8128a63efa534cf29e1d5e22750d38ea: stopping tablet replica
I20260810 00:50:00.522312 23136 raft_consensus.cc:2229] T b55a0fc859004296b58a6f23c9454cbf P 8128a63efa534cf29e1d5e22750d38ea [term 1 LEADER]: Raft consensus shutting down.
I20260810 00:50:00.523293 23137 ts_tablet_manager.cc:1905] T b55a0fc859004296b58a6f23c9454cbf P 4247dc3fa8f34c7bb314fb78609dd96a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:50:00.523360 23136 raft_consensus.cc:2258] T b55a0fc859004296b58a6f23c9454cbf P 8128a63efa534cf29e1d5e22750d38ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:50:00.527362 23136 ts_tablet_manager.cc:1905] T b55a0fc859004296b58a6f23c9454cbf P 8128a63efa534cf29e1d5e22750d38ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:50:00.530004 23137 ts_tablet_manager.cc:1918] T b55a0fc859004296b58a6f23c9454cbf P 4247dc3fa8f34c7bb314fb78609dd96a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:50:00.530231 23137 log.cc:1196] T b55a0fc859004296b58a6f23c9454cbf P 4247dc3fa8f34c7bb314fb78609dd96a: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-1/wal/wals/b55a0fc859004296b58a6f23c9454cbf
I20260810 00:50:00.530913 23137 ts_tablet_manager.cc:1939] T b55a0fc859004296b58a6f23c9454cbf P 4247dc3fa8f34c7bb314fb78609dd96a: Deleting consensus metadata
Aug 10 00:50:00 dist-test-slave-1nl6 krb5kdc[22197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.1@KRBTEST.COM
I20260810 00:50:00.532822 22248 catalog_manager.cc:4841] TS 4247dc3fa8f34c7bb314fb78609dd96a (127.28.6.2:38865): tablet b55a0fc859004296b58a6f23c9454cbf (table test-table [id=c4297862bb0e431c9916268c65e996e4]) successfully deleted
I20260810 00:50:00.535009 23136 ts_tablet_manager.cc:1918] T b55a0fc859004296b58a6f23c9454cbf P 8128a63efa534cf29e1d5e22750d38ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:50:00.535185 23136 log.cc:1196] T b55a0fc859004296b58a6f23c9454cbf P 8128a63efa534cf29e1d5e22750d38ea: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786322687226232-28696-0/minicluster-data/ts-0/wal/wals/b55a0fc859004296b58a6f23c9454cbf
I20260810 00:50:00.535771 23136 ts_tablet_manager.cc:1939] T b55a0fc859004296b58a6f23c9454cbf P 8128a63efa534cf29e1d5e22750d38ea: Deleting consensus metadata
I20260810 00:50:00.537441 22250 catalog_manager.cc:4841] TS 8128a63efa534cf29e1d5e22750d38ea (127.28.6.1:43213): tablet b55a0fc859004296b58a6f23c9454cbf (table test-table [id=c4297862bb0e431c9916268c65e996e4]) successfully deleted
I20260810 00:50:00.539407 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:00.500316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48312 (local address 127.28.6.1:43213)
0810 00:50:00.500498 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:00.500505 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:00.519461 (+ 18956us) server_negotiation.cc:407] Connection header received
0810 00:50:00.519705 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:00.519713 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:00.519829 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:00.519933 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:00.522197 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:00.523509 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:00.525569 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:00.526369 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:00.526500 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:00.532605 (+ 6105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:00.532624 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:00.532644 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:50:00.532704 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:00.536404 (+ 3700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:00.537108 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:00.537117 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:00.537120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:00.537331 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:00.537779 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:00.537785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:00.537788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:00.538154 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:00.538498 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:00.538799 (+ 301us) server_negotiation.cc:299] Negotiation successful
0810 00:50:00.539053 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 00:50:00.540122 22579 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:50:00.637522 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 22377
I20260810 00:50:00.665987 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 22600
Aug 10 00:50:00 dist-test-slave-1nl6 krb5kdc[22197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.2@KRBTEST.COM
I20260810 00:50:00.680414 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:00.666743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49452 (local address 127.28.6.2:38865)
0810 00:50:00.666964 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:00.666969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:00.667050 (+ 81us) server_negotiation.cc:407] Connection header received
0810 00:50:00.667201 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:00.667204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:00.667281 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:00.667366 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:00.668666 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:00.669387 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:00.670574 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:00.671013 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:00.671104 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:00.675229 (+ 4125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:00.675239 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:00.675255 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:50:00.675284 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:00.677990 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:00.678660 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:00.678667 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:00.678670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:00.678795 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:00.679188 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:00.679191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:00.679193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:00.679562 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:00.679792 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:00.680021 (+ 229us) server_negotiation.cc:299] Negotiation successful
0810 00:50:00.680174 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 00:50:00.681070 22806 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:50:00.769007 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 22600
I20260810 00:50:00.797341 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 22824
Aug 10 00:50:00 dist-test-slave-1nl6 krb5kdc[22197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786322997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.3@KRBTEST.COM
I20260810 00:50:00.812264 23084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:00.798145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43448 (local address 127.28.6.3:45715)
0810 00:50:00.798310 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:00.798315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:00.798333 (+ 18us) server_negotiation.cc:407] Connection header received
0810 00:50:00.798497 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:00.798501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:00.798567 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:00.798654 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:00.799985 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:00.800697 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:00.802054 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:00.802446 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:00.802552 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:00.807097 (+ 4545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:00.807113 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:00.807131 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:50:00.807171 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:00.809893 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:00.810467 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:00.810474 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:00.810476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:00.810604 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:00.810940 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:00.810946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:00.810951 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:00.811305 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:00.811629 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:00.811803 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 00:50:00.811969 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 00:50:00.812991 23045 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:50:00.901746 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 22824
I20260810 00:50:00.928820 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 22222
I20260810 00:50:00.930058 22326 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:50:01.039402 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 22222
2026-08-10T00:50:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4800 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-10T00:50:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:50:01Z Disabled control of system clock
I20260810 00:50:01.137916 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:35033
--webserver_interface=127.28.6.62
--webserver_port=0
--builtin_ntp_servers=127.28.6.20:44501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:35033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 00:50:01.474491 23152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:50:01.474751 23152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:50:01.474830 23152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:50:01.481949 23152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:50:01.482045 23152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:50:01.482089 23152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:50:01.482146 23152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:50:01.482189 23152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:50:01.490957 23152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:44501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:35033
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:35033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:50:01.492256 23152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:50:01.493991 23152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:50:01.505589 23164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:01.505546 23167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:01.505618 23163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:50:01.506340 23152 server_base.cc:1034] running on GCE node
I20260810 00:50:01.507125 23152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:50:01.508440 23152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:50:01.509675 23152 hybrid_clock.cc:648] HybridClock initialized: now 1786323001509635 us; error 51 us; skew 500 ppm
I20260810 00:50:01.510023 23152 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 00:50:01.516980 23152 webserver.cc:460] Webserver started at http://127.28.6.62:39933/ using document root <none> and password file <none>
I20260810 00:50:01.517635 23152 fs_manager.cc:359] Metadata directory not provided
I20260810 00:50:01.517763 23152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:50:01.518095 23152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:50:01.521027 23152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "a8e541f2ad4941af84d769d9b042448f"
format_stamp: "Formatted at 2026-08-10 00:50:01 on dist-test-slave-1nl6"
server_key: "31a0461faa7ca196d1c06ae2e5d53947"
server_key_iv: "625af92209a4cf6fabfa8e2f0cf17814"
server_key_version: "encryptionkey@0"
I20260810 00:50:01.521854 23152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "a8e541f2ad4941af84d769d9b042448f"
format_stamp: "Formatted at 2026-08-10 00:50:01 on dist-test-slave-1nl6"
server_key: "31a0461faa7ca196d1c06ae2e5d53947"
server_key_iv: "625af92209a4cf6fabfa8e2f0cf17814"
server_key_version: "encryptionkey@0"
I20260810 00:50:01.527326 23152 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260810 00:50:01.530503 23178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:01.531831 23152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 00:50:01.532011 23152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "a8e541f2ad4941af84d769d9b042448f"
format_stamp: "Formatted at 2026-08-10 00:50:01 on dist-test-slave-1nl6"
server_key: "31a0461faa7ca196d1c06ae2e5d53947"
server_key_iv: "625af92209a4cf6fabfa8e2f0cf17814"
server_key_version: "encryptionkey@0"
I20260810 00:50:01.532249 23152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:50:01.555668 23152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:50:01.556485 23152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:50:01.556785 23152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:50:01.575398 23152 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:35033
I20260810 00:50:01.575434 23253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:35033 every 8 connection(s)
I20260810 00:50:01.576856 23152 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:50:01.580524 23254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:50:01.583214 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 23152
I20260810 00:50:01.583410 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:50:01.583871 28696 external_mini_cluster.cc:1428] Setting key 1b8a6c3580568bbcfbea40c8cfff136d
I20260810 00:50:01.592257 23254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8e541f2ad4941af84d769d9b042448f: Bootstrap starting.
I20260810 00:50:01.596772 23254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8e541f2ad4941af84d769d9b042448f: Neither blocks nor log segments found. Creating new log.
I20260810 00:50:01.598261 23254 log.cc:824] T 00000000000000000000000000000000 P a8e541f2ad4941af84d769d9b042448f: Log is configured to *not* fsync() on all Append() calls
I20260810 00:50:01.599753 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:01.585900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48338 (local address 127.28.6.62:35033)
0810 00:50:01.586889 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:01.586950 (+ 61us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:01.587005 (+ 55us) server_negotiation.cc:407] Connection header received
0810 00:50:01.588616 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:01.588638 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:01.589082 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:01.589491 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:01.590055 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:01.591616 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:01.594012 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:01.595336 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:01.595567 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:01.596965 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:01.596979 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:01.596999 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:01.597542 (+ 543us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:01.597701 (+ 159us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:01.597728 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:01.597896 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:01.598131 (+ 235us) server_negotiation.cc:299] Negotiation successful
0810 00:50:01.598464 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":357,"threads_started":1}
I20260810 00:50:01.602061 23254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8e541f2ad4941af84d769d9b042448f: No bootstrap required, opened a new log
I20260810 00:50:01.614535 23254 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8e541f2ad4941af84d769d9b042448f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8e541f2ad4941af84d769d9b042448f" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 35033 } }
I20260810 00:50:01.615005 23254 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8e541f2ad4941af84d769d9b042448f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:50:01.615123 23254 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8e541f2ad4941af84d769d9b042448f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8e541f2ad4941af84d769d9b042448f, State: Initialized, Role: FOLLOWER
I20260810 00:50:01.615775 23254 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8e541f2ad4941af84d769d9b042448f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8e541f2ad4941af84d769d9b042448f" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 35033 } }
I20260810 00:50:01.616096 23254 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8e541f2ad4941af84d769d9b042448f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:50:01.616250 23254 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8e541f2ad4941af84d769d9b042448f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:50:01.616425 23254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8e541f2ad4941af84d769d9b042448f [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:50:01.619966 23254 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8e541f2ad4941af84d769d9b042448f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8e541f2ad4941af84d769d9b042448f" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 35033 } }
I20260810 00:50:01.620618 23254 leader_election.cc:302] T 00000000000000000000000000000000 P a8e541f2ad4941af84d769d9b042448f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8e541f2ad4941af84d769d9b042448f; no voters:
I20260810 00:50:01.621420 23254 leader_election.cc:288] T 00000000000000000000000000000000 P a8e541f2ad4941af84d769d9b042448f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:50:01.621532 23261 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8e541f2ad4941af84d769d9b042448f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:50:01.623379 23261 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8e541f2ad4941af84d769d9b042448f [term 1 LEADER]: Becoming Leader. State: Replica: a8e541f2ad4941af84d769d9b042448f, State: Running, Role: LEADER
I20260810 00:50:01.623885 23261 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8e541f2ad4941af84d769d9b042448f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8e541f2ad4941af84d769d9b042448f" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 35033 } }
I20260810 00:50:01.626222 23254 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8e541f2ad4941af84d769d9b042448f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:50:01.627090 23262 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8e541f2ad4941af84d769d9b042448f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8e541f2ad4941af84d769d9b042448f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8e541f2ad4941af84d769d9b042448f" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 35033 } } }
I20260810 00:50:01.627482 23262 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8e541f2ad4941af84d769d9b042448f [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:01.632246 23268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:50:01.632441 23263 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8e541f2ad4941af84d769d9b042448f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8e541f2ad4941af84d769d9b042448f. Latest consensus state: current_term: 1 leader_uuid: "a8e541f2ad4941af84d769d9b042448f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8e541f2ad4941af84d769d9b042448f" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 35033 } } }
I20260810 00:50:01.632812 23263 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8e541f2ad4941af84d769d9b042448f [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:01.639083 23268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:50:01.658571 23268 catalog_manager.cc:1347] Generated new cluster ID: 1a6b37507e2143fba89734cee2c22858
I20260810 00:50:01.658761 23268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:50:01.682541 23268 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:50:01.684394 23268 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:50:01.692332 23268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8e541f2ad4941af84d769d9b042448f: Generated new TSK 0
I20260810 00:50:01.693317 23268 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 00:50:01.706735 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.1:0
--local_ip_for_outbound_sockets=127.28.6.1
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:35033
--builtin_ntp_servers=127.28.6.20:44501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 00:50:02.055883 23292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:50:02.056143 23292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:50:02.056284 23292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:50:02.063552 23292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:50:02.063644 23292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:50:02.063825 23292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.1
I20260810 00:50:02.072597 23292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:44501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:35033
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.1
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:50:02.073876 23292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:50:02.075996 23292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:50:02.089097 23308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:02.089133 23305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:02.089188 23301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:50:02.090080 23292 server_base.cc:1034] running on GCE node
I20260810 00:50:02.090595 23292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:50:02.091645 23292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:50:02.092880 23292 hybrid_clock.cc:648] HybridClock initialized: now 1786323002092837 us; error 33 us; skew 500 ppm
I20260810 00:50:02.093155 23292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 00:50:02.095459 23292 webserver.cc:460] Webserver started at http://127.28.6.1:41879/ using document root <none> and password file <none>
I20260810 00:50:02.096071 23292 fs_manager.cc:359] Metadata directory not provided
I20260810 00:50:02.096169 23292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:50:02.096439 23292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:50:02.100020 23292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-0/data/instance:
uuid: "4c0cab705b504426858c8ce376a49786"
format_stamp: "Formatted at 2026-08-10 00:50:02 on dist-test-slave-1nl6"
server_key: "4c48a51b9ec5332400484b4dceb14279"
server_key_iv: "97d32698d9ca6c1952d14439d2b5eef8"
server_key_version: "encryptionkey@0"
I20260810 00:50:02.100889 23292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance:
uuid: "4c0cab705b504426858c8ce376a49786"
format_stamp: "Formatted at 2026-08-10 00:50:02 on dist-test-slave-1nl6"
server_key: "4c48a51b9ec5332400484b4dceb14279"
server_key_iv: "97d32698d9ca6c1952d14439d2b5eef8"
server_key_version: "encryptionkey@0"
I20260810 00:50:02.105835 23292 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260810 00:50:02.109174 23322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:02.110373 23292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 00:50:02.110554 23292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-0/wal
uuid: "4c0cab705b504426858c8ce376a49786"
format_stamp: "Formatted at 2026-08-10 00:50:02 on dist-test-slave-1nl6"
server_key: "4c48a51b9ec5332400484b4dceb14279"
server_key_iv: "97d32698d9ca6c1952d14439d2b5eef8"
server_key_version: "encryptionkey@0"
I20260810 00:50:02.110767 23292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:50:02.133006 23292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:50:02.133821 23292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:50:02.134116 23292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:50:02.135125 23292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:50:02.136794 23292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:50:02.136912 23292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:02.137045 23292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:50:02.137132 23292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:02.182551 23292 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.1:37435
I20260810 00:50:02.182641 23511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.1:37435 every 8 connection(s)
I20260810 00:50:02.183912 23292 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
I20260810 00:50:02.187907 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 23292
I20260810 00:50:02.188094 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance
I20260810 00:50:02.188728 28696 external_mini_cluster.cc:1428] Setting key 66628f31b4ef190e2a626167e49b6853
I20260810 00:50:02.191820 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.2:0
--local_ip_for_outbound_sockets=127.28.6.2
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:35033
--builtin_ntp_servers=127.28.6.20:44501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 00:50:02.197582 23518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:02.186483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:40855 (local address 127.28.6.62:35033)
0810 00:50:02.186975 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:02.186981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:02.188410 (+ 1429us) server_negotiation.cc:407] Connection header received
0810 00:50:02.190655 (+ 2245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:02.190660 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:02.190740 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:02.190828 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:02.192201 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:02.193096 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:02.194623 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:02.195224 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:02.195360 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:02.196732 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:02.196737 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:02.196741 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:02.196758 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:02.196801 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:02.196812 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:02.196911 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:02.197141 (+ 230us) server_negotiation.cc:299] Negotiation successful
0810 00:50:02.197235 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":219,"threads_started":1}
I20260810 00:50:02.198534 23519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:02.187118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:35033 (local address 127.28.6.1:40855)
0810 00:50:02.188237 (+ 1119us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:02.188289 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:02.190361 (+ 2072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:02.191128 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:02.191142 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:02.191256 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:50:02.191975 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:02.192003 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:02.193255 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:02.193261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:02.194473 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:02.194485 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:02.195459 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:02.195468 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:02.195704 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:02.196403 (+ 699us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 00:50:02.196406 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:50:02.196442 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 00:50:02.196519 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:50:02.196913 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:50:02.196928 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:50:02.196937 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 00:50:02.197126 (+ 189us) client_negotiation.cc:240] Negotiation successful
0810 00:50:02.197507 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":290,"threads_started":1}
I20260810 00:50:02.200121 23515 heartbeater.cc:347] Connected to a master server at 127.28.6.62:35033
I20260810 00:50:02.200448 23515 heartbeater.cc:464] Registering TS with master...
I20260810 00:50:02.201311 23515 heartbeater.cc:511] Master 127.28.6.62:35033 requested a full tablet report, sending...
I20260810 00:50:02.203688 23202 ts_manager.cc:194] Registered new tserver with Master: 4c0cab705b504426858c8ce376a49786 (127.28.6.1:37435)
I20260810 00:50:02.205166 23202 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.6.1:40855
W20260810 00:50:02.540736 23523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:50:02.541042 23523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:50:02.541246 23523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:50:02.548377 23523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:50:02.548518 23523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:50:02.548686 23523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.2
I20260810 00:50:02.557580 23523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:44501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:35033
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.2
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:50:02.558804 23523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:50:02.560654 23523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:50:02.573758 23531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:02.573834 23533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:02.573832 23535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:50:02.574877 23523 server_base.cc:1034] running on GCE node
I20260810 00:50:02.575515 23523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:50:02.576501 23523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:50:02.577769 23523 hybrid_clock.cc:648] HybridClock initialized: now 1786323002577701 us; error 55 us; skew 500 ppm
I20260810 00:50:02.578110 23523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 00:50:02.580363 23523 webserver.cc:460] Webserver started at http://127.28.6.2:34245/ using document root <none> and password file <none>
I20260810 00:50:02.581027 23523 fs_manager.cc:359] Metadata directory not provided
I20260810 00:50:02.581139 23523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:50:02.581475 23523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:50:02.584311 23523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-1/data/instance:
uuid: "35f8d4d131a24259b66f5a96d8f76579"
format_stamp: "Formatted at 2026-08-10 00:50:02 on dist-test-slave-1nl6"
server_key: "45ee8add9fcbe2fe4ef28b215710e76d"
server_key_iv: "2846752ad8075451312b2b5e1255b19e"
server_key_version: "encryptionkey@0"
I20260810 00:50:02.585103 23523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance:
uuid: "35f8d4d131a24259b66f5a96d8f76579"
format_stamp: "Formatted at 2026-08-10 00:50:02 on dist-test-slave-1nl6"
server_key: "45ee8add9fcbe2fe4ef28b215710e76d"
server_key_iv: "2846752ad8075451312b2b5e1255b19e"
server_key_version: "encryptionkey@0"
I20260810 00:50:02.590281 23523 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260810 00:50:02.593713 23545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:02.595041 23523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 00:50:02.595214 23523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-1/wal
uuid: "35f8d4d131a24259b66f5a96d8f76579"
format_stamp: "Formatted at 2026-08-10 00:50:02 on dist-test-slave-1nl6"
server_key: "45ee8add9fcbe2fe4ef28b215710e76d"
server_key_iv: "2846752ad8075451312b2b5e1255b19e"
server_key_version: "encryptionkey@0"
I20260810 00:50:02.595422 23523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:50:02.625236 23523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:50:02.626003 23523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:50:02.626350 23523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:50:02.627528 23523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:50:02.628954 23523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:50:02.629071 23523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:02.629204 23523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:50:02.629302 23523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:02.662789 23523 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.2:35051
I20260810 00:50:02.662920 23723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.2:35051 every 8 connection(s)
I20260810 00:50:02.664067 23523 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
I20260810 00:50:02.673363 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 23523
I20260810 00:50:02.673560 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance
I20260810 00:50:02.674033 28696 external_mini_cluster.cc:1428] Setting key 6fc4a0f7b5e1c8d464d8a10b7d3acd47
I20260810 00:50:02.676456 23518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:02.666556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:46547 (local address 127.28.6.62:35033)
0810 00:50:02.666740 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:02.666746 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:02.668111 (+ 1365us) server_negotiation.cc:407] Connection header received
0810 00:50:02.669898 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:02.669904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:02.669994 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:02.670106 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:02.671223 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:02.671958 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:02.673486 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:02.673976 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:02.674105 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:02.675612 (+ 1507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:02.675617 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:02.675620 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:02.675638 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:02.675671 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:02.675684 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:02.675762 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:02.676062 (+ 300us) server_negotiation.cc:299] Negotiation successful
0810 00:50:02.676155 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 00:50:02.677145 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.3:0
--local_ip_for_outbound_sockets=127.28.6.3
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:35033
--builtin_ntp_servers=127.28.6.20:44501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 00:50:02.677423 23727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:02.667089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:35033 (local address 127.28.6.2:46547)
0810 00:50:02.667953 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:02.667994 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:02.669628 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:02.670340 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:02.670348 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:02.670428 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:50:02.671057 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:02.671077 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:02.672119 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:02.672125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:02.673338 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:02.673351 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:02.674204 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:02.674211 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:02.674436 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:02.675286 (+ 850us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 00:50:02.675290 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:50:02.675338 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 00:50:02.675424 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:50:02.675797 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:50:02.675807 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:50:02.675815 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 00:50:02.676013 (+ 198us) client_negotiation.cc:240] Negotiation successful
0810 00:50:02.676350 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":275,"threads_started":1}
I20260810 00:50:02.678799 23724 heartbeater.cc:347] Connected to a master server at 127.28.6.62:35033
I20260810 00:50:02.679152 23724 heartbeater.cc:464] Registering TS with master...
I20260810 00:50:02.680061 23724 heartbeater.cc:511] Master 127.28.6.62:35033 requested a full tablet report, sending...
I20260810 00:50:02.681545 23202 ts_manager.cc:194] Registered new tserver with Master: 35f8d4d131a24259b66f5a96d8f76579 (127.28.6.2:35051)
I20260810 00:50:02.682345 23202 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.6.2:46547
W20260810 00:50:03.014405 23729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:50:03.014662 23729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:50:03.014832 23729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:50:03.021909 23729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:50:03.022044 23729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:50:03.022228 23729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.3
I20260810 00:50:03.031563 23729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:44501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:35033
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.3
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:50:03.032852 23729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:50:03.034668 23729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:50:03.048345 23738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:03.048838 23737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:03.048902 23740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:50:03.049319 23729 server_base.cc:1034] running on GCE node
I20260810 00:50:03.050107 23729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:50:03.051102 23729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:50:03.052343 23729 hybrid_clock.cc:648] HybridClock initialized: now 1786323003052300 us; error 40 us; skew 500 ppm
I20260810 00:50:03.052676 23729 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 00:50:03.055116 23729 webserver.cc:460] Webserver started at http://127.28.6.3:37245/ using document root <none> and password file <none>
I20260810 00:50:03.055830 23729 fs_manager.cc:359] Metadata directory not provided
I20260810 00:50:03.055976 23729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:50:03.056313 23729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:50:03.059309 23729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-2/data/instance:
uuid: "9c75f70e66224f81afe7e8a0dab7dc92"
format_stamp: "Formatted at 2026-08-10 00:50:03 on dist-test-slave-1nl6"
server_key: "af17e0390c92dc1f29d1b4e9a49abff6"
server_key_iv: "fdda6524fd09d7aa4a1ee2ecb0d0a904"
server_key_version: "encryptionkey@0"
I20260810 00:50:03.060127 23729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance:
uuid: "9c75f70e66224f81afe7e8a0dab7dc92"
format_stamp: "Formatted at 2026-08-10 00:50:03 on dist-test-slave-1nl6"
server_key: "af17e0390c92dc1f29d1b4e9a49abff6"
server_key_iv: "fdda6524fd09d7aa4a1ee2ecb0d0a904"
server_key_version: "encryptionkey@0"
I20260810 00:50:03.065675 23729 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260810 00:50:03.069128 23755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:03.070384 23729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 00:50:03.070567 23729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-2/wal
uuid: "9c75f70e66224f81afe7e8a0dab7dc92"
format_stamp: "Formatted at 2026-08-10 00:50:03 on dist-test-slave-1nl6"
server_key: "af17e0390c92dc1f29d1b4e9a49abff6"
server_key_iv: "fdda6524fd09d7aa4a1ee2ecb0d0a904"
server_key_version: "encryptionkey@0"
I20260810 00:50:03.070783 23729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:50:03.101387 23729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:50:03.102223 23729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:50:03.102509 23729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:50:03.103577 23729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:50:03.105075 23729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:50:03.105209 23729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:03.105340 23729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:50:03.105436 23729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:03.139657 23729 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.3:37391
I20260810 00:50:03.139744 23932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.3:37391 every 8 connection(s)
I20260810 00:50:03.140986 23729 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
I20260810 00:50:03.149706 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 23729
I20260810 00:50:03.149888 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance
I20260810 00:50:03.150368 28696 external_mini_cluster.cc:1428] Setting key 853dca1326b8f63503fb9ec38eb095dc
I20260810 00:50:03.153679 23518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:03.143632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:57513 (local address 127.28.6.62:35033)
0810 00:50:03.143826 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:03.143831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:03.145234 (+ 1403us) server_negotiation.cc:407] Connection header received
0810 00:50:03.146987 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:03.146992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:03.147081 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:03.147193 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:03.148234 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:03.148981 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:03.150638 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:03.151177 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:03.151320 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:03.152926 (+ 1606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:03.152934 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:03.152938 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:03.152960 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:03.153000 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:03.153017 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:03.153141 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:03.153364 (+ 223us) server_negotiation.cc:299] Negotiation successful
0810 00:50:03.153437 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 00:50:03.154697 23935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:03.144196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:35033 (local address 127.28.6.3:57513)
0810 00:50:03.145076 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:03.145121 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:03.146712 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:03.147438 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:03.147445 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:03.147522 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:50:03.148070 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:03.148086 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:03.149134 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:03.149143 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:03.150487 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:03.150502 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:03.151457 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:03.151468 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:03.151802 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:03.152631 (+ 829us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 00:50:03.152633 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:50:03.152670 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 00:50:03.152739 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:50:03.153138 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:50:03.153152 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:50:03.153162 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 00:50:03.153356 (+ 194us) client_negotiation.cc:240] Negotiation successful
0810 00:50:03.153655 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":295,"threads_started":1}
I20260810 00:50:03.155975 23933 heartbeater.cc:347] Connected to a master server at 127.28.6.62:35033
I20260810 00:50:03.156263 23933 heartbeater.cc:464] Registering TS with master...
I20260810 00:50:03.156989 23933 heartbeater.cc:511] Master 127.28.6.62:35033 requested a full tablet report, sending...
I20260810 00:50:03.158349 23202 ts_manager.cc:194] Registered new tserver with Master: 9c75f70e66224f81afe7e8a0dab7dc92 (127.28.6.3:37391)
I20260810 00:50:03.159169 23202 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.6.3:57513
I20260810 00:50:03.165342 28696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 00:50:03.175096 23518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:03.169460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48348 (local address 127.28.6.62:35033)
0810 00:50:03.169642 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:03.169647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:03.170093 (+ 446us) server_negotiation.cc:407] Connection header received
0810 00:50:03.170232 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:03.170236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:03.170302 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:03.170386 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:03.170816 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:03.171570 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:03.172980 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:03.173387 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:03.173483 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:03.174437 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:03.174442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:03.174446 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:03.174462 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:03.174492 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:03.174503 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:03.174570 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:03.174699 (+ 129us) server_negotiation.cc:299] Negotiation successful
0810 00:50:03.174763 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 00:50:03.178643 23202 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48348:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 00:50:03.185494 23202 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 00:50:03.207404 23950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:03.196688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:35051 (local address 127.0.0.1:44024)
0810 00:50:03.198130 (+ 1442us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:03.198180 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:03.198353 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:03.200073 (+ 1720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:03.200084 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:03.200127 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:50:03.200660 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:03.200676 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:03.202928 (+ 2252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:03.202935 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:03.205008 (+ 2073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:03.205024 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:03.206379 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:03.206386 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:03.206642 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:03.206679 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 00:50:03.206819 (+ 140us) client_negotiation.cc:240] Negotiation successful
0810 00:50:03.207042 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1156,"mutex_wait_us":2,"thread_start_us":982,"threads_started":1}
I20260810 00:50:03.208220 23515 heartbeater.cc:503] Master 127.28.6.62:35033 was elected leader, sending a full tablet report...
I20260810 00:50:03.208676 23952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:03.196951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44024 (local address 127.28.6.2:35051)
0810 00:50:03.199307 (+ 2356us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:03.199317 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:03.199352 (+ 35us) server_negotiation.cc:407] Connection header received
0810 00:50:03.199487 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:03.199497 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:03.199763 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:03.200968 (+ 1205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:50:03.201225 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:03.202758 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:03.205217 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:03.206157 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:03.206315 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:03.206574 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:03.207047 (+ 473us) server_negotiation.cc:299] Negotiation successful
0810 00:50:03.207223 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2194,"thread_start_us":2049,"threads_started":1}
I20260810 00:50:03.209069 23953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:03.197142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:37435 (local address 127.0.0.1:33664)
0810 00:50:03.199047 (+ 1905us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:03.199072 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:03.199188 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:03.201717 (+ 2529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:03.201721 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:03.201753 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:50:03.202124 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:03.202142 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:03.203879 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:03.203884 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:03.206387 (+ 2503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:03.206402 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:03.208353 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:03.208359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:03.208531 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:03.208564 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 00:50:03.208637 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 00:50:03.208773 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1690,"thread_start_us":1684,"threads_started":1}
I20260810 00:50:03.209620 23957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:03.199234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:37391 (local address 127.0.0.1:59952)
0810 00:50:03.201156 (+ 1922us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:03.201183 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:03.201290 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:03.202062 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:03.202066 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:03.202100 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:50:03.202481 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:03.202489 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:03.204429 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:03.204436 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:03.206647 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:03.206659 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:03.208774 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:03.208782 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:03.208975 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:03.209023 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 00:50:03.209107 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 00:50:03.209233 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1778,"mutex_wait_us":25,"thread_start_us":1625,"threads_started":1}
I20260810 00:50:03.210215 23959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:03.199732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59952 (local address 127.28.6.3:37391)
0810 00:50:03.201508 (+ 1776us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:03.201516 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:03.201549 (+ 33us) server_negotiation.cc:407] Connection header received
0810 00:50:03.201642 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:03.201651 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:03.201859 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:03.202042 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:50:03.202658 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:03.204271 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:03.206842 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:03.208235 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:03.208372 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:03.208621 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:03.209711 (+ 1090us) server_negotiation.cc:299] Negotiation successful
0810 00:50:03.209884 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1584,"thread_start_us":1424,"threads_started":1}
I20260810 00:50:03.211277 23954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:03.197457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33664 (local address 127.28.6.1:37435)
0810 00:50:03.198798 (+ 1341us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:03.198807 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:03.199274 (+ 467us) server_negotiation.cc:407] Connection header received
0810 00:50:03.199381 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:03.199392 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:03.201542 (+ 2150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:03.201728 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:50:03.202336 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:03.203724 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:03.206630 (+ 2906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:03.208115 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:03.210394 (+ 2279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:03.210660 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:03.210738 (+ 78us) server_negotiation.cc:299] Negotiation successful
0810 00:50:03.210883 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1171,"thread_start_us":993,"threads_started":1}
I20260810 00:50:03.213483 23619 tablet_service.cc:1467] Processing CreateTablet for tablet 817707acbacc4fe882a2192e8616e241 (DEFAULT_TABLE table=test-table [id=7157866586564a8691fd97c6247daf03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:50:03.213480 23829 tablet_service.cc:1467] Processing CreateTablet for tablet 817707acbacc4fe882a2192e8616e241 (DEFAULT_TABLE table=test-table [id=7157866586564a8691fd97c6247daf03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:50:03.214955 23619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 817707acbacc4fe882a2192e8616e241. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:50:03.215004 23829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 817707acbacc4fe882a2192e8616e241. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:50:03.214599 23400 tablet_service.cc:1467] Processing CreateTablet for tablet 817707acbacc4fe882a2192e8616e241 (DEFAULT_TABLE table=test-table [id=7157866586564a8691fd97c6247daf03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:50:03.215906 23400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 817707acbacc4fe882a2192e8616e241. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:50:03.227316 23960 tablet_bootstrap.cc:492] T 817707acbacc4fe882a2192e8616e241 P 35f8d4d131a24259b66f5a96d8f76579: Bootstrap starting.
I20260810 00:50:03.229952 23961 tablet_bootstrap.cc:492] T 817707acbacc4fe882a2192e8616e241 P 9c75f70e66224f81afe7e8a0dab7dc92: Bootstrap starting.
I20260810 00:50:03.230810 23966 tablet_bootstrap.cc:492] T 817707acbacc4fe882a2192e8616e241 P 4c0cab705b504426858c8ce376a49786: Bootstrap starting.
I20260810 00:50:03.231488 23960 tablet_bootstrap.cc:654] T 817707acbacc4fe882a2192e8616e241 P 35f8d4d131a24259b66f5a96d8f76579: Neither blocks nor log segments found. Creating new log.
I20260810 00:50:03.232842 23960 log.cc:824] T 817707acbacc4fe882a2192e8616e241 P 35f8d4d131a24259b66f5a96d8f76579: Log is configured to *not* fsync() on all Append() calls
I20260810 00:50:03.233623 23961 tablet_bootstrap.cc:654] T 817707acbacc4fe882a2192e8616e241 P 9c75f70e66224f81afe7e8a0dab7dc92: Neither blocks nor log segments found. Creating new log.
I20260810 00:50:03.233707 23966 tablet_bootstrap.cc:654] T 817707acbacc4fe882a2192e8616e241 P 4c0cab705b504426858c8ce376a49786: Neither blocks nor log segments found. Creating new log.
I20260810 00:50:03.234979 23961 log.cc:824] T 817707acbacc4fe882a2192e8616e241 P 9c75f70e66224f81afe7e8a0dab7dc92: Log is configured to *not* fsync() on all Append() calls
I20260810 00:50:03.235067 23966 log.cc:824] T 817707acbacc4fe882a2192e8616e241 P 4c0cab705b504426858c8ce376a49786: Log is configured to *not* fsync() on all Append() calls
I20260810 00:50:03.236304 23960 tablet_bootstrap.cc:492] T 817707acbacc4fe882a2192e8616e241 P 35f8d4d131a24259b66f5a96d8f76579: No bootstrap required, opened a new log
I20260810 00:50:03.236755 23960 ts_tablet_manager.cc:1397] T 817707acbacc4fe882a2192e8616e241 P 35f8d4d131a24259b66f5a96d8f76579: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260810 00:50:03.238104 23961 tablet_bootstrap.cc:492] T 817707acbacc4fe882a2192e8616e241 P 9c75f70e66224f81afe7e8a0dab7dc92: No bootstrap required, opened a new log
I20260810 00:50:03.238377 23961 ts_tablet_manager.cc:1397] T 817707acbacc4fe882a2192e8616e241 P 9c75f70e66224f81afe7e8a0dab7dc92: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260810 00:50:03.239050 23966 tablet_bootstrap.cc:492] T 817707acbacc4fe882a2192e8616e241 P 4c0cab705b504426858c8ce376a49786: No bootstrap required, opened a new log
I20260810 00:50:03.239308 23966 ts_tablet_manager.cc:1397] T 817707acbacc4fe882a2192e8616e241 P 4c0cab705b504426858c8ce376a49786: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260810 00:50:03.246131 23961 raft_consensus.cc:348] T 817707acbacc4fe882a2192e8616e241 P 9c75f70e66224f81afe7e8a0dab7dc92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f8d4d131a24259b66f5a96d8f76579" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35051 } } peers { permanent_uuid: "4c0cab705b504426858c8ce376a49786" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 37435 } } peers { permanent_uuid: "9c75f70e66224f81afe7e8a0dab7dc92" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 37391 } }
I20260810 00:50:03.246511 23961 raft_consensus.cc:374] T 817707acbacc4fe882a2192e8616e241 P 9c75f70e66224f81afe7e8a0dab7dc92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:50:03.246647 23961 raft_consensus.cc:729] T 817707acbacc4fe882a2192e8616e241 P 9c75f70e66224f81afe7e8a0dab7dc92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c75f70e66224f81afe7e8a0dab7dc92, State: Initialized, Role: FOLLOWER
I20260810 00:50:03.247304 23961 consensus_queue.cc:260] T 817707acbacc4fe882a2192e8616e241 P 9c75f70e66224f81afe7e8a0dab7dc92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f8d4d131a24259b66f5a96d8f76579" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35051 } } peers { permanent_uuid: "4c0cab705b504426858c8ce376a49786" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 37435 } } peers { permanent_uuid: "9c75f70e66224f81afe7e8a0dab7dc92" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 37391 } }
I20260810 00:50:03.247349 23960 raft_consensus.cc:348] T 817707acbacc4fe882a2192e8616e241 P 35f8d4d131a24259b66f5a96d8f76579 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f8d4d131a24259b66f5a96d8f76579" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35051 } } peers { permanent_uuid: "4c0cab705b504426858c8ce376a49786" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 37435 } } peers { permanent_uuid: "9c75f70e66224f81afe7e8a0dab7dc92" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 37391 } }
I20260810 00:50:03.247772 23960 raft_consensus.cc:374] T 817707acbacc4fe882a2192e8616e241 P 35f8d4d131a24259b66f5a96d8f76579 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:50:03.247870 23960 raft_consensus.cc:729] T 817707acbacc4fe882a2192e8616e241 P 35f8d4d131a24259b66f5a96d8f76579 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35f8d4d131a24259b66f5a96d8f76579, State: Initialized, Role: FOLLOWER
I20260810 00:50:03.248397 23960 consensus_queue.cc:260] T 817707acbacc4fe882a2192e8616e241 P 35f8d4d131a24259b66f5a96d8f76579 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f8d4d131a24259b66f5a96d8f76579" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35051 } } peers { permanent_uuid: "4c0cab705b504426858c8ce376a49786" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 37435 } } peers { permanent_uuid: "9c75f70e66224f81afe7e8a0dab7dc92" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 37391 } }
I20260810 00:50:03.249693 23724 heartbeater.cc:503] Master 127.28.6.62:35033 was elected leader, sending a full tablet report...
I20260810 00:50:03.249543 23966 raft_consensus.cc:348] T 817707acbacc4fe882a2192e8616e241 P 4c0cab705b504426858c8ce376a49786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f8d4d131a24259b66f5a96d8f76579" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35051 } } peers { permanent_uuid: "4c0cab705b504426858c8ce376a49786" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 37435 } } peers { permanent_uuid: "9c75f70e66224f81afe7e8a0dab7dc92" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 37391 } }
I20260810 00:50:03.249925 23966 raft_consensus.cc:374] T 817707acbacc4fe882a2192e8616e241 P 4c0cab705b504426858c8ce376a49786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:50:03.250043 23966 raft_consensus.cc:729] T 817707acbacc4fe882a2192e8616e241 P 4c0cab705b504426858c8ce376a49786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c0cab705b504426858c8ce376a49786, State: Initialized, Role: FOLLOWER
I20260810 00:50:03.250275 23960 ts_tablet_manager.cc:1428] T 817707acbacc4fe882a2192e8616e241 P 35f8d4d131a24259b66f5a96d8f76579: Time spent starting tablet: real 0.013s user 0.004s sys 0.008s
I20260810 00:50:03.250604 23966 consensus_queue.cc:260] T 817707acbacc4fe882a2192e8616e241 P 4c0cab705b504426858c8ce376a49786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f8d4d131a24259b66f5a96d8f76579" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35051 } } peers { permanent_uuid: "4c0cab705b504426858c8ce376a49786" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 37435 } } peers { permanent_uuid: "9c75f70e66224f81afe7e8a0dab7dc92" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 37391 } }
I20260810 00:50:03.251187 23933 heartbeater.cc:503] Master 127.28.6.62:35033 was elected leader, sending a full tablet report...
I20260810 00:50:03.253168 23966 ts_tablet_manager.cc:1428] T 817707acbacc4fe882a2192e8616e241 P 4c0cab705b504426858c8ce376a49786: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260810 00:50:03.253398 23961 ts_tablet_manager.cc:1428] T 817707acbacc4fe882a2192e8616e241 P 9c75f70e66224f81afe7e8a0dab7dc92: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
W20260810 00:50:03.392817 23934 tablet.cc:2369] T 817707acbacc4fe882a2192e8616e241 P 9c75f70e66224f81afe7e8a0dab7dc92: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 00:50:03.416162 23725 tablet.cc:2369] T 817707acbacc4fe882a2192e8616e241 P 35f8d4d131a24259b66f5a96d8f76579: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 00:50:03.436241 23516 tablet.cc:2369] T 817707acbacc4fe882a2192e8616e241 P 4c0cab705b504426858c8ce376a49786: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:50:03.586511 23974 raft_consensus.cc:482] T 817707acbacc4fe882a2192e8616e241 P 4c0cab705b504426858c8ce376a49786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:50:03.586838 23974 raft_consensus.cc:504] T 817707acbacc4fe882a2192e8616e241 P 4c0cab705b504426858c8ce376a49786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f8d4d131a24259b66f5a96d8f76579" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35051 } } peers { permanent_uuid: "4c0cab705b504426858c8ce376a49786" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 37435 } } peers { permanent_uuid: "9c75f70e66224f81afe7e8a0dab7dc92" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 37391 } }
I20260810 00:50:03.588595 23974 leader_election.cc:288] T 817707acbacc4fe882a2192e8616e241 P 4c0cab705b504426858c8ce376a49786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35f8d4d131a24259b66f5a96d8f76579 (127.28.6.2:35051), 9c75f70e66224f81afe7e8a0dab7dc92 (127.28.6.3:37391)
I20260810 00:50:03.594975 23959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:03.589010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:35987 (local address 127.28.6.3:37391)
0810 00:50:03.589209 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:03.589218 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:03.589701 (+ 483us) server_negotiation.cc:407] Connection header received
0810 00:50:03.589945 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:03.589949 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:03.590011 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:03.590108 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:50:03.590728 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:03.591501 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:03.593338 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:03.594048 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:03.594181 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:03.594287 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:03.594572 (+ 285us) server_negotiation.cc:299] Negotiation successful
0810 00:50:03.594694 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 00:50:03.595057 23975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:03.588873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:37391 (local address 127.28.6.1:35987)
0810 00:50:03.589574 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:03.589593 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:03.589780 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:03.590129 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:03.590134 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:03.590160 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:50:03.590566 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:03.590575 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:03.591628 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:03.591634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:03.593095 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:03.593108 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:03.594245 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:03.594251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:03.594401 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:03.594447 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 00:50:03.594544 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 00:50:03.594668 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"spinlock_wait_cycles":18176,"thread_start_us":459,"threads_started":1}
I20260810 00:50:03.595538 23952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:03.589088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:46025 (local address 127.28.6.2:35051)
0810 00:50:03.589312 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:03.589321 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:03.590216 (+ 895us) server_negotiation.cc:407] Connection header received
0810 00:50:03.590423 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:03.590430 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:03.590509 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:03.590605 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:50:03.591211 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:03.591918 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:03.593740 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:03.594504 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:03.594649 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:03.594745 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:03.595105 (+ 360us) server_negotiation.cc:299] Negotiation successful
0810 00:50:03.595240 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 00:50:03.596035 23976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:03.589009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:35051 (local address 127.28.6.1:46025)
0810 00:50:03.590103 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:03.590118 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:03.590272 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:03.590637 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:03.590641 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:03.590664 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:50:03.591042 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:03.591052 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:03.592070 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:03.592076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:03.593562 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:03.593575 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:03.594705 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:03.594711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:03.594865 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:03.594888 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 00:50:03.595716 (+ 828us) client_negotiation.cc:240] Negotiation successful
0810 00:50:03.595831 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":994,"thread_start_us":718,"threads_started":1}
I20260810 00:50:03.596091 23859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "817707acbacc4fe882a2192e8616e241" candidate_uuid: "4c0cab705b504426858c8ce376a49786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c75f70e66224f81afe7e8a0dab7dc92" is_pre_election: true
I20260810 00:50:03.596647 23859 raft_consensus.cc:2454] T 817707acbacc4fe882a2192e8616e241 P 9c75f70e66224f81afe7e8a0dab7dc92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c0cab705b504426858c8ce376a49786 in term 0.
I20260810 00:50:03.597050 23660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "817707acbacc4fe882a2192e8616e241" candidate_uuid: "4c0cab705b504426858c8ce376a49786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35f8d4d131a24259b66f5a96d8f76579" is_pre_election: true
I20260810 00:50:03.597334 23327 leader_election.cc:302] T 817707acbacc4fe882a2192e8616e241 P 4c0cab705b504426858c8ce376a49786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c0cab705b504426858c8ce376a49786, 9c75f70e66224f81afe7e8a0dab7dc92; no voters:
I20260810 00:50:03.597569 23660 raft_consensus.cc:2454] T 817707acbacc4fe882a2192e8616e241 P 35f8d4d131a24259b66f5a96d8f76579 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c0cab705b504426858c8ce376a49786 in term 0.
I20260810 00:50:03.597772 23974 raft_consensus.cc:2781] T 817707acbacc4fe882a2192e8616e241 P 4c0cab705b504426858c8ce376a49786 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 00:50:03.597929 23974 raft_consensus.cc:482] T 817707acbacc4fe882a2192e8616e241 P 4c0cab705b504426858c8ce376a49786 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 00:50:03.598065 23974 raft_consensus.cc:3037] T 817707acbacc4fe882a2192e8616e241 P 4c0cab705b504426858c8ce376a49786 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:50:03.600375 23974 raft_consensus.cc:504] T 817707acbacc4fe882a2192e8616e241 P 4c0cab705b504426858c8ce376a49786 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f8d4d131a24259b66f5a96d8f76579" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35051 } } peers { permanent_uuid: "4c0cab705b504426858c8ce376a49786" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 37435 } } peers { permanent_uuid: "9c75f70e66224f81afe7e8a0dab7dc92" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 37391 } }
I20260810 00:50:03.600942 23974 leader_election.cc:288] T 817707acbacc4fe882a2192e8616e241 P 4c0cab705b504426858c8ce376a49786 [CANDIDATE]: Term 1 election: Requested vote from peers 35f8d4d131a24259b66f5a96d8f76579 (127.28.6.2:35051), 9c75f70e66224f81afe7e8a0dab7dc92 (127.28.6.3:37391)
I20260810 00:50:03.601451 23660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "817707acbacc4fe882a2192e8616e241" candidate_uuid: "4c0cab705b504426858c8ce376a49786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35f8d4d131a24259b66f5a96d8f76579"
I20260810 00:50:03.601501 23859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "817707acbacc4fe882a2192e8616e241" candidate_uuid: "4c0cab705b504426858c8ce376a49786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c75f70e66224f81afe7e8a0dab7dc92"
I20260810 00:50:03.601679 23660 raft_consensus.cc:3037] T 817707acbacc4fe882a2192e8616e241 P 35f8d4d131a24259b66f5a96d8f76579 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:50:03.601686 23859 raft_consensus.cc:3037] T 817707acbacc4fe882a2192e8616e241 P 9c75f70e66224f81afe7e8a0dab7dc92 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:50:03.604945 23660 raft_consensus.cc:2454] T 817707acbacc4fe882a2192e8616e241 P 35f8d4d131a24259b66f5a96d8f76579 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c0cab705b504426858c8ce376a49786 in term 1.
I20260810 00:50:03.604945 23859 raft_consensus.cc:2454] T 817707acbacc4fe882a2192e8616e241 P 9c75f70e66224f81afe7e8a0dab7dc92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c0cab705b504426858c8ce376a49786 in term 1.
I20260810 00:50:03.605434 23327 leader_election.cc:302] T 817707acbacc4fe882a2192e8616e241 P 4c0cab705b504426858c8ce376a49786 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c0cab705b504426858c8ce376a49786, 9c75f70e66224f81afe7e8a0dab7dc92; no voters:
I20260810 00:50:03.605691 23974 raft_consensus.cc:2781] T 817707acbacc4fe882a2192e8616e241 P 4c0cab705b504426858c8ce376a49786 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:50:03.606141 23974 raft_consensus.cc:686] T 817707acbacc4fe882a2192e8616e241 P 4c0cab705b504426858c8ce376a49786 [term 1 LEADER]: Becoming Leader. State: Replica: 4c0cab705b504426858c8ce376a49786, State: Running, Role: LEADER
I20260810 00:50:03.606581 23974 consensus_queue.cc:237] T 817707acbacc4fe882a2192e8616e241 P 4c0cab705b504426858c8ce376a49786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f8d4d131a24259b66f5a96d8f76579" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35051 } } peers { permanent_uuid: "4c0cab705b504426858c8ce376a49786" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 37435 } } peers { permanent_uuid: "9c75f70e66224f81afe7e8a0dab7dc92" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 37391 } }
I20260810 00:50:03.611284 23202 catalog_manager.cc:5495] T 817707acbacc4fe882a2192e8616e241 P 4c0cab705b504426858c8ce376a49786 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c0cab705b504426858c8ce376a49786 (127.28.6.1). New cstate: current_term: 1 leader_uuid: "4c0cab705b504426858c8ce376a49786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f8d4d131a24259b66f5a96d8f76579" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c0cab705b504426858c8ce376a49786" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 37435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c75f70e66224f81afe7e8a0dab7dc92" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 37391 } health_report { overall_health: UNKNOWN } } }
I20260810 00:50:03.657913 23954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:03.652221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33680 (local address 127.28.6.1:37435)
0810 00:50:03.652404 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:03.652412 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:03.652436 (+ 24us) server_negotiation.cc:407] Connection header received
0810 00:50:03.652541 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:03.652544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:03.652633 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:03.652815 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:50:03.653420 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:03.654220 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:03.655824 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:03.656275 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:03.656380 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:03.656833 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:50:03.657326 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:50:03.657450 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:03.657567 (+ 117us) server_negotiation.cc:299] Negotiation successful
0810 00:50:03.657629 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 00:50:03.667251 23660 raft_consensus.cc:1261] T 817707acbacc4fe882a2192e8616e241 P 35f8d4d131a24259b66f5a96d8f76579 [term 1 FOLLOWER]: Refusing update from remote peer 4c0cab705b504426858c8ce376a49786: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:50:03.667379 23859 raft_consensus.cc:1261] T 817707acbacc4fe882a2192e8616e241 P 9c75f70e66224f81afe7e8a0dab7dc92 [term 1 FOLLOWER]: Refusing update from remote peer 4c0cab705b504426858c8ce376a49786: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:50:03.668316 23974 consensus_queue.cc:1035] T 817707acbacc4fe882a2192e8616e241 P 4c0cab705b504426858c8ce376a49786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35f8d4d131a24259b66f5a96d8f76579" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 35051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:50:03.668640 23977 consensus_queue.cc:1035] T 817707acbacc4fe882a2192e8616e241 P 4c0cab705b504426858c8ce376a49786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c75f70e66224f81afe7e8a0dab7dc92" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 37391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:50:03.688802 23985 mvcc.cc:205] Tried to move back new op lower bound from 7316779023009529856 to 7316779022776012800. Current Snapshot: MvccSnapshot[applied={T|T < 7316779023009529856}]
I20260810 00:50:03.698426 23202 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:50:03.698689 23202 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:50:03.702215 23202 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a8e541f2ad4941af84d769d9b042448f: Sending DeleteTablet for 3 replicas of tablet 817707acbacc4fe882a2192e8616e241
I20260810 00:50:03.703331 23619 tablet_service.cc:1514] Processing DeleteTablet for tablet 817707acbacc4fe882a2192e8616e241 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:50:03 UTC) from {username='slave'} at 127.0.0.1:44024
I20260810 00:50:03.703449 23400 tablet_service.cc:1514] Processing DeleteTablet for tablet 817707acbacc4fe882a2192e8616e241 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:50:03 UTC) from {username='slave'} at 127.0.0.1:33664
I20260810 00:50:03.704125 23829 tablet_service.cc:1514] Processing DeleteTablet for tablet 817707acbacc4fe882a2192e8616e241 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:50:03 UTC) from {username='slave'} at 127.0.0.1:59952
I20260810 00:50:03.704963 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 23292
I20260810 00:50:03.706890 23999 tablet_replica.cc:330] T 817707acbacc4fe882a2192e8616e241 P 9c75f70e66224f81afe7e8a0dab7dc92: stopping tablet replica
I20260810 00:50:03.707541 23999 raft_consensus.cc:2229] T 817707acbacc4fe882a2192e8616e241 P 9c75f70e66224f81afe7e8a0dab7dc92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:50:03.708190 23999 raft_consensus.cc:2258] T 817707acbacc4fe882a2192e8616e241 P 9c75f70e66224f81afe7e8a0dab7dc92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:50:03.710414 23999 ts_tablet_manager.cc:1905] T 817707acbacc4fe882a2192e8616e241 P 9c75f70e66224f81afe7e8a0dab7dc92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:50:03.714973 23997 tablet_replica.cc:330] T 817707acbacc4fe882a2192e8616e241 P 35f8d4d131a24259b66f5a96d8f76579: stopping tablet replica
I20260810 00:50:03.715679 23997 raft_consensus.cc:2229] T 817707acbacc4fe882a2192e8616e241 P 35f8d4d131a24259b66f5a96d8f76579 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:50:03.716295 23999 ts_tablet_manager.cc:1918] T 817707acbacc4fe882a2192e8616e241 P 9c75f70e66224f81afe7e8a0dab7dc92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:50:03.716387 23997 raft_consensus.cc:2258] T 817707acbacc4fe882a2192e8616e241 P 35f8d4d131a24259b66f5a96d8f76579 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:50:03.716543 23999 log.cc:1196] T 817707acbacc4fe882a2192e8616e241 P 9c75f70e66224f81afe7e8a0dab7dc92: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-2/wal/wals/817707acbacc4fe882a2192e8616e241
I20260810 00:50:03.717283 23996 tablet_replica.cc:330] T 817707acbacc4fe882a2192e8616e241 P 4c0cab705b504426858c8ce376a49786: stopping tablet replica
I20260810 00:50:03.717731 23996 raft_consensus.cc:2229] T 817707acbacc4fe882a2192e8616e241 P 4c0cab705b504426858c8ce376a49786 [term 1 LEADER]: Raft consensus shutting down.
I20260810 00:50:03.718468 23996 raft_consensus.cc:2258] T 817707acbacc4fe882a2192e8616e241 P 4c0cab705b504426858c8ce376a49786 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:50:03.718986 23997 ts_tablet_manager.cc:1905] T 817707acbacc4fe882a2192e8616e241 P 35f8d4d131a24259b66f5a96d8f76579: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:50:03.720500 23999 ts_tablet_manager.cc:1939] T 817707acbacc4fe882a2192e8616e241 P 9c75f70e66224f81afe7e8a0dab7dc92: Deleting consensus metadata
I20260810 00:50:03.720533 23996 ts_tablet_manager.cc:1905] T 817707acbacc4fe882a2192e8616e241 P 4c0cab705b504426858c8ce376a49786: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:50:03.722316 23182 catalog_manager.cc:4841] TS 9c75f70e66224f81afe7e8a0dab7dc92 (127.28.6.3:37391): tablet 817707acbacc4fe882a2192e8616e241 (table test-table [id=7157866586564a8691fd97c6247daf03]) successfully deleted
I20260810 00:50:03.725095 23954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:03.706246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33690 (local address 127.28.6.1:37435)
0810 00:50:03.706466 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:03.706477 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:03.714148 (+ 7671us) server_negotiation.cc:407] Connection header received
0810 00:50:03.714481 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:03.714487 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:03.714586 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:03.714659 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:03.715497 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:03.716666 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:03.718670 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:03.719376 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:03.719480 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:03.723121 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:03.723133 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:03.723152 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:03.723925 (+ 773us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:03.723984 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:03.724009 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:03.724109 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:03.724506 (+ 397us) server_negotiation.cc:299] Negotiation successful
0810 00:50:03.724665 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 00:50:03.726135 23502 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:50:03.726826 23997 ts_tablet_manager.cc:1918] T 817707acbacc4fe882a2192e8616e241 P 35f8d4d131a24259b66f5a96d8f76579: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:50:03.727077 23997 log.cc:1196] T 817707acbacc4fe882a2192e8616e241 P 35f8d4d131a24259b66f5a96d8f76579: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786322687226232-28696-0/minicluster-data/ts-1/wal/wals/817707acbacc4fe882a2192e8616e241
I20260810 00:50:03.727842 23997 ts_tablet_manager.cc:1939] T 817707acbacc4fe882a2192e8616e241 P 35f8d4d131a24259b66f5a96d8f76579: Deleting consensus metadata
I20260810 00:50:03.730053 23179 catalog_manager.cc:4841] TS 35f8d4d131a24259b66f5a96d8f76579 (127.28.6.2:35051): tablet 817707acbacc4fe882a2192e8616e241 (table test-table [id=7157866586564a8691fd97c6247daf03]) successfully deleted
I20260810 00:50:03.821197 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 23292
I20260810 00:50:03.847595 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 23523
W20260810 00:50:03.847568 23182 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.6.1:37435 (error 108)
I20260810 00:50:03.850358 24011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:03.849563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:37435 (local address 127.0.0.1:33704)
0810 00:50:03.850040 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:03.850175 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.1:37435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":264,"threads_started":1}
W20260810 00:50:03.850658 23182 catalog_manager.cc:4584] TS 4c0cab705b504426858c8ce376a49786 (127.28.6.1:37435): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 817707acbacc4fe882a2192e8616e241: Network error: Client connection negotiation failed: client connection to 127.28.6.1:37435: connect: Connection refused (error 111)
I20260810 00:50:03.855114 23952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:03.848461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44036 (local address 127.28.6.2:35051)
0810 00:50:03.848680 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:03.848686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:03.848965 (+ 279us) server_negotiation.cc:407] Connection header received
0810 00:50:03.849182 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:03.849203 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:03.849327 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:03.849448 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:03.849949 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:03.850790 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:03.852163 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:03.852557 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:03.852652 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:03.853611 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:03.853620 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:03.853634 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:03.854220 (+ 586us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:03.854267 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:03.854287 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:03.854380 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:03.854484 (+ 104us) server_negotiation.cc:299] Negotiation successful
0810 00:50:03.854568 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 00:50:03.855754 23719 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:50:03.904255 24011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:03.903716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:37435 (local address 127.0.0.1:33712)
0810 00:50:03.903942 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:03.904042 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.1:37435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260810 00:50:03.905380 24011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:03.904992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:37435 (local address 127.0.0.1:33728)
0810 00:50:03.905156 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:03.905234 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.1:37435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260810 00:50:03.945874 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 23523
I20260810 00:50:03.966182 24011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:03.965710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:37435 (local address 127.0.0.1:33740)
0810 00:50:03.965930 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:03.966035 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.1:37435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260810 00:50:03.967300 24011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:03.966933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:37435 (local address 127.0.0.1:33742)
0810 00:50:03.967093 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:03.967161 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.1:37435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 00:50:03.974731 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 23729
I20260810 00:50:03.981099 23959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:03.975552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59954 (local address 127.28.6.3:37391)
0810 00:50:03.975717 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:03.975722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:03.975740 (+ 18us) server_negotiation.cc:407] Connection header received
0810 00:50:03.975901 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:03.975905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:03.975994 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:03.976076 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:03.976435 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:03.977140 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:03.978312 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:03.978711 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:03.978803 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:03.979806 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:03.979816 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:03.979832 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:03.980394 (+ 562us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:03.980442 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:03.980462 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:03.980549 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:03.980665 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 00:50:03.980793 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 00:50:03.981734 23923 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:50:04.048043 24011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:04.047473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:37435 (local address 127.0.0.1:33754)
0810 00:50:04.047727 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:04.047852 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.1:37435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260810 00:50:04.049260 24011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:04.048753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:37435 (local address 127.0.0.1:33760)
0810 00:50:04.048983 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:04.049085 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.1:37435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260810 00:50:04.070130 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 23729
I20260810 00:50:04.095193 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 23152
I20260810 00:50:04.096242 23248 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:50:04.206100 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 23152
2026-08-10T00:50:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3164 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:50:04 dist-test-slave-1nl6 krb5kdc[24021](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:50:04 dist-test-slave-1nl6 krb5kdc[24021](info): set up 2 sockets
Aug 10 00:50:04 dist-test-slave-1nl6 krb5kdc[24021](info): commencing operation
krb5kdc: starting...
W20260810 00:50:05.522328 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.264s user 0.004s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:50:05 dist-test-slave-1nl6 krb5kdc[24021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:50:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:50:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.6.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
I20260810 00:50:05.795576 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:35357
--webserver_interface=127.28.6.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:38433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:35357
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:50:06.143895 24041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:50:06.144425 24041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:50:06.144641 24041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:50:06.152967 24041 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:50:06.153113 24041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:50:06.153223 24041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:50:06.153313 24041 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:50:06.153380 24041 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:50:06.163447 24041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:38433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:35357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:35357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:50:06.164803 24041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:50:06.166189 24041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:50:06.177937 24048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:06.178066 24047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:06.178036 24053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:50:06.179253 24041 server_base.cc:1034] running on GCE node
I20260810 00:50:06.180126 24041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:50:06.181458 24041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:50:06.182821 24041 hybrid_clock.cc:648] HybridClock initialized: now 1786323006182775 us; error 50 us; skew 500 ppm
I20260810 00:50:06.183233 24041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:50:06 dist-test-slave-1nl6 krb5kdc[24021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323006, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:50:06.186345 24041 init.cc:377] Logged in from keytab as kudu/127.28.6.62@KRBTEST.COM (short username kudu)
I20260810 00:50:06.187912 24041 webserver.cc:460] Webserver started at http://127.28.6.62:40991/ using document root <none> and password file <none>
I20260810 00:50:06.188612 24041 fs_manager.cc:359] Metadata directory not provided
I20260810 00:50:06.188764 24041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:50:06.189083 24041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:50:06.192085 24041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "e037c4be8b814040847febe7761ec66c"
format_stamp: "Formatted at 2026-08-10 00:50:06 on dist-test-slave-1nl6"
server_key: "967c878d23b680b86c8f4991b0fd6f88"
server_key_iv: "ea498d453ea362028c0f0d46db653db7"
server_key_version: "encryptionkey@0"
I20260810 00:50:06.192862 24041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "e037c4be8b814040847febe7761ec66c"
format_stamp: "Formatted at 2026-08-10 00:50:06 on dist-test-slave-1nl6"
server_key: "967c878d23b680b86c8f4991b0fd6f88"
server_key_iv: "ea498d453ea362028c0f0d46db653db7"
server_key_version: "encryptionkey@0"
I20260810 00:50:06.197894 24041 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260810 00:50:06.201225 24064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:06.202414 24041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 00:50:06.202606 24041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "e037c4be8b814040847febe7761ec66c"
format_stamp: "Formatted at 2026-08-10 00:50:06 on dist-test-slave-1nl6"
server_key: "967c878d23b680b86c8f4991b0fd6f88"
server_key_iv: "ea498d453ea362028c0f0d46db653db7"
server_key_version: "encryptionkey@0"
I20260810 00:50:06.202800 24041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:50:06.227397 24041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:50:06.231734 24041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:50:06.232004 24041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:50:06.250545 24041 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:35357
I20260810 00:50:06.250582 24141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:35357 every 8 connection(s)
I20260810 00:50:06.251888 24041 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:50:06.255541 24142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:50:06.261546 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 24041
I20260810 00:50:06.261734 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:50:06.262195 28696 external_mini_cluster.cc:1428] Setting key bc56ada7099caa9246a563bb9ad745a2
I20260810 00:50:06.265947 24142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e037c4be8b814040847febe7761ec66c: Bootstrap starting.
I20260810 00:50:06.269745 24142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e037c4be8b814040847febe7761ec66c: Neither blocks nor log segments found. Creating new log.
I20260810 00:50:06.270784 24142 log.cc:824] T 00000000000000000000000000000000 P e037c4be8b814040847febe7761ec66c: Log is configured to *not* fsync() on all Append() calls
I20260810 00:50:06.273823 24142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e037c4be8b814040847febe7761ec66c: No bootstrap required, opened a new log
Aug 10 00:50:06 dist-test-slave-1nl6 krb5kdc[24021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:50:06.282135 24142 raft_consensus.cc:348] T 00000000000000000000000000000000 P e037c4be8b814040847febe7761ec66c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e037c4be8b814040847febe7761ec66c" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 35357 } }
I20260810 00:50:06.282505 24142 raft_consensus.cc:374] T 00000000000000000000000000000000 P e037c4be8b814040847febe7761ec66c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:50:06.282617 24142 raft_consensus.cc:729] T 00000000000000000000000000000000 P e037c4be8b814040847febe7761ec66c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e037c4be8b814040847febe7761ec66c, State: Initialized, Role: FOLLOWER
I20260810 00:50:06.283227 24142 consensus_queue.cc:260] T 00000000000000000000000000000000 P e037c4be8b814040847febe7761ec66c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e037c4be8b814040847febe7761ec66c" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 35357 } }
I20260810 00:50:06.283483 24142 raft_consensus.cc:388] T 00000000000000000000000000000000 P e037c4be8b814040847febe7761ec66c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:50:06.283648 24142 raft_consensus.cc:482] T 00000000000000000000000000000000 P e037c4be8b814040847febe7761ec66c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:50:06.283855 24142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e037c4be8b814040847febe7761ec66c [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:50:06.285758 24146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:06.264129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53906 (local address 127.28.6.62:35357)
0810 00:50:06.264943 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:06.264954 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:06.264996 (+ 42us) server_negotiation.cc:407] Connection header received
0810 00:50:06.266709 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:06.266733 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:06.267149 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:06.267463 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:06.269387 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:06.270729 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:06.272478 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:06.273116 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:06.273245 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:06.278056 (+ 4811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:06.278064 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:06.278077 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:50:06.278120 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:06.281286 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:06.281994 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:06.282001 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:06.282007 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:06.282174 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:06.282587 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:06.282593 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:06.282598 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:06.283219 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:06.283551 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:06.283939 (+ 388us) server_negotiation.cc:299] Negotiation successful
0810 00:50:06.284374 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":394,"threads_started":1}
I20260810 00:50:06.286837 24142 raft_consensus.cc:504] T 00000000000000000000000000000000 P e037c4be8b814040847febe7761ec66c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e037c4be8b814040847febe7761ec66c" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 35357 } }
I20260810 00:50:06.287443 24142 leader_election.cc:302] T 00000000000000000000000000000000 P e037c4be8b814040847febe7761ec66c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e037c4be8b814040847febe7761ec66c; no voters:
I20260810 00:50:06.288096 24142 leader_election.cc:288] T 00000000000000000000000000000000 P e037c4be8b814040847febe7761ec66c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:50:06.288264 24149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e037c4be8b814040847febe7761ec66c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:50:06.289346 24149 raft_consensus.cc:686] T 00000000000000000000000000000000 P e037c4be8b814040847febe7761ec66c [term 1 LEADER]: Becoming Leader. State: Replica: e037c4be8b814040847febe7761ec66c, State: Running, Role: LEADER
I20260810 00:50:06.289919 24149 consensus_queue.cc:237] T 00000000000000000000000000000000 P e037c4be8b814040847febe7761ec66c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e037c4be8b814040847febe7761ec66c" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 35357 } }
I20260810 00:50:06.290347 24142 sys_catalog.cc:564] T 00000000000000000000000000000000 P e037c4be8b814040847febe7761ec66c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:50:06.292753 24151 sys_catalog.cc:455] T 00000000000000000000000000000000 P e037c4be8b814040847febe7761ec66c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e037c4be8b814040847febe7761ec66c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e037c4be8b814040847febe7761ec66c" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 35357 } } }
I20260810 00:50:06.293110 24151 sys_catalog.cc:458] T 00000000000000000000000000000000 P e037c4be8b814040847febe7761ec66c [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:06.295387 24152 sys_catalog.cc:455] T 00000000000000000000000000000000 P e037c4be8b814040847febe7761ec66c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e037c4be8b814040847febe7761ec66c. Latest consensus state: current_term: 1 leader_uuid: "e037c4be8b814040847febe7761ec66c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e037c4be8b814040847febe7761ec66c" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 35357 } } }
I20260810 00:50:06.295644 24152 sys_catalog.cc:458] T 00000000000000000000000000000000 P e037c4be8b814040847febe7761ec66c [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:06.296386 24156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:50:06.302578 24156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:50:06.311479 24156 catalog_manager.cc:1347] Generated new cluster ID: 82f9e3327e624756b7b60bb70c0ccc80
I20260810 00:50:06.311585 24156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:50:06.329995 24156 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:50:06.331820 24156 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:50:06.346762 24156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e037c4be8b814040847febe7761ec66c: Generated new TSK 0
I20260810 00:50:06.347570 24156 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.6.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab.
Entry for principal kudu/127.28.6.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab.
I20260810 00:50:06.431373 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.1:0
--local_ip_for_outbound_sockets=127.28.6.1
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:35357
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:38433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:50:06.772867 24182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:50:06.773168 24182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:50:06.773342 24182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:50:06.780315 24182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:50:06.780409 24182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:50:06.780524 24182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.1
I20260810 00:50:06.789731 24182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:38433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:35357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.1
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:50:06.790992 24182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:50:06.792323 24182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:50:06.805529 24188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:06.805554 24191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:06.805903 24194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:50:06.806558 24182 server_base.cc:1034] running on GCE node
I20260810 00:50:06.807204 24182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:50:06.808070 24182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:50:06.809265 24182 hybrid_clock.cc:648] HybridClock initialized: now 1786323006809223 us; error 30 us; skew 500 ppm
I20260810 00:50:06.809554 24182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:50:06 dist-test-slave-1nl6 krb5kdc[24021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323006, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:50:06.817493 24182 init.cc:377] Logged in from keytab as kudu/127.28.6.1@KRBTEST.COM (short username kudu)
I20260810 00:50:06.818854 24182 webserver.cc:460] Webserver started at http://127.28.6.1:41915/ using document root <none> and password file <none>
I20260810 00:50:06.819545 24182 fs_manager.cc:359] Metadata directory not provided
I20260810 00:50:06.819674 24182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:50:06.819989 24182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:50:06.822798 24182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-0/data/instance:
uuid: "18d6502671b14f558df3565d1e54169e"
format_stamp: "Formatted at 2026-08-10 00:50:06 on dist-test-slave-1nl6"
server_key: "c148631e1605ee0b76463af0fc07d1ed"
server_key_iv: "da2d77b29b352af117b4162f25b45250"
server_key_version: "encryptionkey@0"
I20260810 00:50:06.823583 24182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance:
uuid: "18d6502671b14f558df3565d1e54169e"
format_stamp: "Formatted at 2026-08-10 00:50:06 on dist-test-slave-1nl6"
server_key: "c148631e1605ee0b76463af0fc07d1ed"
server_key_iv: "da2d77b29b352af117b4162f25b45250"
server_key_version: "encryptionkey@0"
I20260810 00:50:06.828415 24182 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 00:50:06.831601 24205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:06.832837 24182 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 00:50:06.833012 24182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-0/wal
uuid: "18d6502671b14f558df3565d1e54169e"
format_stamp: "Formatted at 2026-08-10 00:50:06 on dist-test-slave-1nl6"
server_key: "c148631e1605ee0b76463af0fc07d1ed"
server_key_iv: "da2d77b29b352af117b4162f25b45250"
server_key_version: "encryptionkey@0"
I20260810 00:50:06.833204 24182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:50:06.849306 24182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:50:06.853453 24182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:50:06.853744 24182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:50:06.854636 24182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:50:06.855957 24182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:50:06.856068 24182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:06.856220 24182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:50:06.856293 24182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:06.887428 24182 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.1:41673
I20260810 00:50:06.887513 24376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.1:41673 every 8 connection(s)
I20260810 00:50:06.888690 24182 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
I20260810 00:50:06.893069 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 24182
I20260810 00:50:06.893375 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance
I20260810 00:50:06.894033 28696 external_mini_cluster.cc:1428] Setting key eb6249343c2fc4215c6c10dad62dfbc7
Aug 10 00:50:06 dist-test-slave-1nl6 krb5kdc[24021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323006, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:50:06.910929 24379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:06.891177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:41807 (local address 127.28.6.62:35357)
0810 00:50:06.891673 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:06.891694 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:06.892619 (+ 925us) server_negotiation.cc:407] Connection header received
0810 00:50:06.895218 (+ 2599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:06.895223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:06.895321 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:06.895444 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:06.897314 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:06.898075 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:06.900620 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:06.901004 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:06.901111 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:06.905108 (+ 3997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:06.905113 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:06.905116 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:50:06.905148 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:06.907837 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:06.908577 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:06.908582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:06.908584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:06.908706 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:06.909164 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:06.909168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:06.909170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:06.909426 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:06.909599 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:06.910310 (+ 711us) server_negotiation.cc:299] Negotiation successful
0810 00:50:06.910544 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":284,"threads_started":1}
I20260810 00:50:06.911784 24381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:06.891710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:35357 (local address 127.28.6.1:41807)
0810 00:50:06.892464 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:06.892500 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:06.894890 (+ 2390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:06.895684 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:06.895692 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:06.896356 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:50:06.897128 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:06.897156 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:06.898244 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:06.898254 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:06.900482 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:06.900499 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:06.901229 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:06.901239 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:06.901562 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:06.902262 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:50:06.902304 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:50:06.904892 (+ 2588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:50:06.907985 (+ 3093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:50:06.907994 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:50:06.908008 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:50:06.908435 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:50:06.908822 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:50:06.908827 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:50:06.908830 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:50:06.909028 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:50:06.909572 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:50:06.909580 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:50:06.909904 (+ 324us) client_negotiation.cc:769] Sending connection context
0810 00:50:06.910173 (+ 269us) client_negotiation.cc:240] Negotiation successful
0810 00:50:06.910662 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":251,"threads_started":1}
I20260810 00:50:06.913518 24377 heartbeater.cc:347] Connected to a master server at 127.28.6.62:35357
I20260810 00:50:06.913865 24377 heartbeater.cc:464] Registering TS with master...
I20260810 00:50:06.914577 24377 heartbeater.cc:511] Master 127.28.6.62:35357 requested a full tablet report, sending...
I20260810 00:50:06.916926 24094 ts_manager.cc:194] Registered new tserver with Master: 18d6502671b14f558df3565d1e54169e (127.28.6.1:41673)
I20260810 00:50:06.919188 24094 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.1@KRBTEST.COM'} at 127.28.6.1:41807
WARNING: no policy specified for kudu/127.28.6.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab.
Entry for principal kudu/127.28.6.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab.
I20260810 00:50:06.959865 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.2:0
--local_ip_for_outbound_sockets=127.28.6.2
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:35357
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:38433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:50:07.296000 24384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:50:07.296315 24384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:50:07.296490 24384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:50:07.303778 24384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:50:07.303933 24384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:50:07.304136 24384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.2
I20260810 00:50:07.313429 24384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:38433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:35357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.2
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:50:07.314628 24384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:50:07.315945 24384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:50:07.328794 24396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:07.328930 24393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:07.328900 24394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:50:07.329809 24384 server_base.cc:1034] running on GCE node
I20260810 00:50:07.330367 24384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:50:07.331344 24384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:50:07.332542 24384 hybrid_clock.cc:648] HybridClock initialized: now 1786323007332484 us; error 57 us; skew 500 ppm
I20260810 00:50:07.332813 24384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:50:07 dist-test-slave-1nl6 krb5kdc[24021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323007, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:50:07.336112 24384 init.cc:377] Logged in from keytab as kudu/127.28.6.2@KRBTEST.COM (short username kudu)
I20260810 00:50:07.337831 24384 webserver.cc:460] Webserver started at http://127.28.6.2:41627/ using document root <none> and password file <none>
I20260810 00:50:07.338536 24384 fs_manager.cc:359] Metadata directory not provided
I20260810 00:50:07.338672 24384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:50:07.339120 24384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:50:07.342144 24384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-1/data/instance:
uuid: "b61978fb2e8841cdb64c820d6dc0bb2b"
format_stamp: "Formatted at 2026-08-10 00:50:07 on dist-test-slave-1nl6"
server_key: "b37208f297261568470a85f11d3be024"
server_key_iv: "6b81ffe5aadec516bca36036aa96aba3"
server_key_version: "encryptionkey@0"
I20260810 00:50:07.342972 24384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance:
uuid: "b61978fb2e8841cdb64c820d6dc0bb2b"
format_stamp: "Formatted at 2026-08-10 00:50:07 on dist-test-slave-1nl6"
server_key: "b37208f297261568470a85f11d3be024"
server_key_iv: "6b81ffe5aadec516bca36036aa96aba3"
server_key_version: "encryptionkey@0"
I20260810 00:50:07.348115 24384 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260810 00:50:07.351413 24413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:07.352684 24384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 00:50:07.352819 24384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-1/wal
uuid: "b61978fb2e8841cdb64c820d6dc0bb2b"
format_stamp: "Formatted at 2026-08-10 00:50:07 on dist-test-slave-1nl6"
server_key: "b37208f297261568470a85f11d3be024"
server_key_iv: "6b81ffe5aadec516bca36036aa96aba3"
server_key_version: "encryptionkey@0"
I20260810 00:50:07.352984 24384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:50:07.374889 24384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:50:07.379482 24384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:50:07.379722 24384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:50:07.380666 24384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:50:07.381970 24384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:50:07.382050 24384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:07.382156 24384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:50:07.382200 24384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:07.417047 24384 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.2:45219
I20260810 00:50:07.417136 24588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.2:45219 every 8 connection(s)
I20260810 00:50:07.418375 24384 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
I20260810 00:50:07.426020 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 24384
I20260810 00:50:07.426196 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance
I20260810 00:50:07.426685 28696 external_mini_cluster.cc:1428] Setting key 995822d8bd0c3f426d20afdb3711ca0e
Aug 10 00:50:07 dist-test-slave-1nl6 krb5kdc[24021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323007, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:50:07.441819 24592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:07.420950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:34131 (local address 127.28.6.62:35357)
0810 00:50:07.421404 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:07.421409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:07.422360 (+ 951us) server_negotiation.cc:407] Connection header received
0810 00:50:07.424313 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:07.424319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:07.424421 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:07.424546 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:07.426049 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:07.427026 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:07.428815 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:07.429480 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:07.429609 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:07.434857 (+ 5248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:07.434862 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:07.434866 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:50:07.434948 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:07.438660 (+ 3712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:07.439438 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:07.439445 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:07.439448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:07.439604 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:07.440070 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:07.440076 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:07.440079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:07.440354 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:07.440514 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:07.441214 (+ 700us) server_negotiation.cc:299] Negotiation successful
0810 00:50:07.441448 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":240,"threads_started":1}
I20260810 00:50:07.442806 24593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:07.421480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:35357 (local address 127.28.6.2:34131)
0810 00:50:07.422191 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:07.422236 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:07.424036 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:07.424800 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:07.424808 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:07.425333 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:50:07.425869 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:07.425887 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:07.427187 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:07.427192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:07.428667 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:07.428684 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:07.429961 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:07.429969 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:07.430346 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:07.431215 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:50:07.431251 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:50:07.434610 (+ 3359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:50:07.438803 (+ 4193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:50:07.438812 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:50:07.438829 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:50:07.439301 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:50:07.439719 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:50:07.439724 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:50:07.439728 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:50:07.439941 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:50:07.440490 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:50:07.440497 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:50:07.440817 (+ 320us) client_negotiation.cc:769] Sending connection context
0810 00:50:07.441087 (+ 270us) client_negotiation.cc:240] Negotiation successful
0810 00:50:07.441606 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":272,"threads_started":1}
I20260810 00:50:07.444401 24590 heartbeater.cc:347] Connected to a master server at 127.28.6.62:35357
I20260810 00:50:07.444728 24590 heartbeater.cc:464] Registering TS with master...
I20260810 00:50:07.445574 24590 heartbeater.cc:511] Master 127.28.6.62:35357 requested a full tablet report, sending...
I20260810 00:50:07.447216 24094 ts_manager.cc:194] Registered new tserver with Master: b61978fb2e8841cdb64c820d6dc0bb2b (127.28.6.2:45219)
I20260810 00:50:07.448474 24094 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.2@KRBTEST.COM'} at 127.28.6.2:34131
WARNING: no policy specified for kudu/127.28.6.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab.
Entry for principal kudu/127.28.6.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab.
I20260810 00:50:07.493950 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.3:0
--local_ip_for_outbound_sockets=127.28.6.3
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:35357
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:38433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:50:07.831883 24598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:50:07.832137 24598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:50:07.832370 24598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:50:07.840348 24598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:50:07.840479 24598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:50:07.840643 24598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.3
I20260810 00:50:07.850018 24598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:38433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:35357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.3
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:50:07.851477 24598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:50:07.852787 24598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:50:07.866106 24612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:07.866056 24606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:07.866058 24605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:50:07.866952 24598 server_base.cc:1034] running on GCE node
I20260810 00:50:07.867515 24598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:50:07.868377 24598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:50:07.869626 24598 hybrid_clock.cc:648] HybridClock initialized: now 1786323007869578 us; error 56 us; skew 500 ppm
I20260810 00:50:07.869946 24598 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:50:07 dist-test-slave-1nl6 krb5kdc[24021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323007, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:50:07.873188 24598 init.cc:377] Logged in from keytab as kudu/127.28.6.3@KRBTEST.COM (short username kudu)
I20260810 00:50:07.874562 24598 webserver.cc:460] Webserver started at http://127.28.6.3:44771/ using document root <none> and password file <none>
I20260810 00:50:07.875291 24598 fs_manager.cc:359] Metadata directory not provided
I20260810 00:50:07.875453 24598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:50:07.875788 24598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:50:07.878579 24598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-2/data/instance:
uuid: "d9cccd0afb46475c9ef17a7001a15c3c"
format_stamp: "Formatted at 2026-08-10 00:50:07 on dist-test-slave-1nl6"
server_key: "2adcf5b9e59e02c29be16b6caa701a68"
server_key_iv: "0b580ad8aa2e250f9654f5471e9a7a42"
server_key_version: "encryptionkey@0"
I20260810 00:50:07.879379 24598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance:
uuid: "d9cccd0afb46475c9ef17a7001a15c3c"
format_stamp: "Formatted at 2026-08-10 00:50:07 on dist-test-slave-1nl6"
server_key: "2adcf5b9e59e02c29be16b6caa701a68"
server_key_iv: "0b580ad8aa2e250f9654f5471e9a7a42"
server_key_version: "encryptionkey@0"
I20260810 00:50:07.884335 24598 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260810 00:50:07.888682 24622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:07.890245 24598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 00:50:07.890407 24598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-2/wal
uuid: "d9cccd0afb46475c9ef17a7001a15c3c"
format_stamp: "Formatted at 2026-08-10 00:50:07 on dist-test-slave-1nl6"
server_key: "2adcf5b9e59e02c29be16b6caa701a68"
server_key_iv: "0b580ad8aa2e250f9654f5471e9a7a42"
server_key_version: "encryptionkey@0"
I20260810 00:50:07.890621 24598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:50:07.910101 24598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:50:07.914202 24598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:50:07.914453 24598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:50:07.915369 24598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:50:07.916571 24598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:50:07.916644 24598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:07.916741 24598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:50:07.916786 24598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:07.922012 24377 heartbeater.cc:503] Master 127.28.6.62:35357 was elected leader, sending a full tablet report...
I20260810 00:50:07.950824 24598 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.3:39925
I20260810 00:50:07.950898 24806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.3:39925 every 8 connection(s)
I20260810 00:50:07.952191 24598 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
I20260810 00:50:07.961823 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 24598
I20260810 00:50:07.962037 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance
I20260810 00:50:07.962575 28696 external_mini_cluster.cc:1428] Setting key 00f6df93cfb428e8b1cb4146805a3042
Aug 10 00:50:07 dist-test-slave-1nl6 krb5kdc[24021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323007, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:50:07.971912 24811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:07.954686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:48919 (local address 127.28.6.62:35357)
0810 00:50:07.955203 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:07.955208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:07.956024 (+ 816us) server_negotiation.cc:407] Connection header received
0810 00:50:07.958019 (+ 1995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:07.958024 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:07.958127 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:07.958250 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:07.959773 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:07.960538 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:07.962514 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:07.963016 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:07.963120 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:07.966548 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:07.966553 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:07.966557 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:50:07.966588 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:07.969239 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:07.969873 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:07.969877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:07.969879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:07.969995 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:07.970368 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:07.970371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:07.970373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:07.970619 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:07.970733 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:07.971456 (+ 723us) server_negotiation.cc:299] Negotiation successful
0810 00:50:07.971637 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":290,"threads_started":1}
I20260810 00:50:07.972736 24812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:07.955116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:35357 (local address 127.28.6.3:48919)
0810 00:50:07.955860 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:07.955914 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:07.957751 (+ 1837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:07.958504 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:07.958511 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:07.959061 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:50:07.959600 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:07.959617 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:07.960698 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:07.960704 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:07.962359 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:07.962376 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:07.963273 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:07.963283 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:07.963572 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:07.964280 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:50:07.964318 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:50:07.966375 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:50:07.969361 (+ 2986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:50:07.969367 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:50:07.969378 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:50:07.969775 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:50:07.970088 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:50:07.970091 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:50:07.970093 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:50:07.970254 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:50:07.970726 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:50:07.970732 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:50:07.971088 (+ 356us) client_negotiation.cc:769] Sending connection context
0810 00:50:07.971311 (+ 223us) client_negotiation.cc:240] Negotiation successful
0810 00:50:07.971660 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":277,"threads_started":1}
I20260810 00:50:07.973995 24809 heartbeater.cc:347] Connected to a master server at 127.28.6.62:35357
I20260810 00:50:07.974315 24809 heartbeater.cc:464] Registering TS with master...
I20260810 00:50:07.975086 24809 heartbeater.cc:511] Master 127.28.6.62:35357 requested a full tablet report, sending...
I20260810 00:50:07.976765 24094 ts_manager.cc:194] Registered new tserver with Master: d9cccd0afb46475c9ef17a7001a15c3c (127.28.6.3:39925)
I20260810 00:50:07.977882 24094 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.3@KRBTEST.COM'} at 127.28.6.3:48919
I20260810 00:50:07.978102 28696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 00:50:07.995827 24811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:07.982217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57538 (local address 127.28.6.62:35357)
0810 00:50:07.982413 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:07.982422 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:07.982812 (+ 390us) server_negotiation.cc:407] Connection header received
0810 00:50:07.983032 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:07.983039 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:07.983103 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:07.983220 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:07.984664 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:07.985362 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:07.986735 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:07.987233 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:07.987417 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:07.990305 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:07.990311 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:07.990315 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:50:07.990355 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:07.992963 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:07.993750 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:07.993755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:07.993765 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:07.993878 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:07.994353 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:07.994358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:07.994361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:07.994634 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:07.994898 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:07.995340 (+ 442us) server_negotiation.cc:299] Negotiation successful
0810 00:50:07.995538 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 00:50:07.999802 24094 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57538:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 00:50:08.006822 24094 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 00:50:08.032195 24829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:08.019191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:39925 (local address 127.0.0.1:53164)
0810 00:50:08.019943 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:08.019992 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:08.020174 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:08.021439 (+ 1265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:08.021451 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:08.021504 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:50:08.022058 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:08.022075 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:08.024771 (+ 2696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:08.024779 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:08.027990 (+ 3211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:08.028013 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:08.030932 (+ 2919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:08.030942 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:08.031259 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:08.031352 (+ 93us) client_negotiation.cc:769] Sending connection context
0810 00:50:08.031517 (+ 165us) client_negotiation.cc:240] Negotiation successful
0810 00:50:08.031711 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"mutex_wait_us":1,"thread_start_us":1077,"threads_started":1}
I20260810 00:50:08.033030 24830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:08.019532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53164 (local address 127.28.6.3:39925)
0810 00:50:08.020420 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:08.020429 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:08.020465 (+ 36us) server_negotiation.cc:407] Connection header received
0810 00:50:08.020569 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:08.020580 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:08.020851 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:08.021066 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:50:08.022349 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:08.024554 (+ 2205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:08.028242 (+ 3688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:08.030243 (+ 2001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:08.030416 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:08.030731 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:08.032462 (+ 1731us) server_negotiation.cc:299] Negotiation successful
0810 00:50:08.032667 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":484,"threads_started":1}
I20260810 00:50:08.037281 24705 tablet_service.cc:1467] Processing CreateTablet for tablet ca4e72d29df945d28c32a34ab50cade0 (DEFAULT_TABLE table=test-table [id=ca93f315a9d243ea999240114a5d290c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:50:08.039259 24705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca4e72d29df945d28c32a34ab50cade0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:50:08.043732 24837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:08.021119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39078 (local address 127.28.6.1:41673)
0810 00:50:08.031937 (+ 10818us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:08.031946 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:08.033639 (+ 1693us) server_negotiation.cc:407] Connection header received
0810 00:50:08.033785 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:08.033796 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:08.034094 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:08.034346 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:50:08.035100 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:08.036712 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:08.039816 (+ 3104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:08.041256 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:08.041388 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:08.041623 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:08.043247 (+ 1624us) server_negotiation.cc:299] Negotiation successful
0810 00:50:08.043411 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10615,"thread_start_us":10829,"threads_started":1}
I20260810 00:50:08.044248 24836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:08.020847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:41673 (local address 127.0.0.1:39078)
0810 00:50:08.032900 (+ 12053us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:08.032919 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:08.033046 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:08.034340 (+ 1294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:08.034345 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:08.034377 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:50:08.034784 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:08.034799 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:08.036893 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:08.036898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:08.039624 (+ 2726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:08.039642 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:08.041780 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:08.041786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:08.042025 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:08.042064 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 00:50:08.042146 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 00:50:08.042277 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11908,"mutex_wait_us":32,"thread_start_us":22837,"threads_started":1}
I20260810 00:50:08.046017 24829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:08.020905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:45219 (local address 127.0.0.1:42858)
0810 00:50:08.033312 (+ 12407us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:08.033333 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:08.033443 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:08.037211 (+ 3768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:08.037219 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:08.037273 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:50:08.037680 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:08.037698 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:08.040222 (+ 2524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:08.040229 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:08.043335 (+ 3106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:08.043357 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:08.045252 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:08.045257 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:08.045428 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:08.045459 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 00:50:08.045582 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 00:50:08.045712 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12269,"mutex_wait_us":1,"thread_start_us":11702,"threads_started":1}
I20260810 00:50:08.046099 24839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:08.021462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42858 (local address 127.28.6.2:45219)
0810 00:50:08.034883 (+ 13421us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:08.034894 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:08.034949 (+ 55us) server_negotiation.cc:407] Connection header received
0810 00:50:08.035064 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:08.035074 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:08.035339 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:08.035549 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:50:08.037923 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:08.039995 (+ 2072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:08.043925 (+ 3930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:08.045083 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:08.045247 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:08.045444 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:08.045626 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 00:50:08.045776 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13214,"thread_start_us":10808,"threads_started":1}
I20260810 00:50:08.048172 24278 tablet_service.cc:1467] Processing CreateTablet for tablet ca4e72d29df945d28c32a34ab50cade0 (DEFAULT_TABLE table=test-table [id=ca93f315a9d243ea999240114a5d290c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:50:08.048918 24487 tablet_service.cc:1467] Processing CreateTablet for tablet ca4e72d29df945d28c32a34ab50cade0 (DEFAULT_TABLE table=test-table [id=ca93f315a9d243ea999240114a5d290c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:50:08.049674 24278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca4e72d29df945d28c32a34ab50cade0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:50:08.050239 24487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca4e72d29df945d28c32a34ab50cade0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:50:08.054605 24840 tablet_bootstrap.cc:492] T ca4e72d29df945d28c32a34ab50cade0 P d9cccd0afb46475c9ef17a7001a15c3c: Bootstrap starting.
I20260810 00:50:08.058838 24840 tablet_bootstrap.cc:654] T ca4e72d29df945d28c32a34ab50cade0 P d9cccd0afb46475c9ef17a7001a15c3c: Neither blocks nor log segments found. Creating new log.
I20260810 00:50:08.060298 24840 log.cc:824] T ca4e72d29df945d28c32a34ab50cade0 P d9cccd0afb46475c9ef17a7001a15c3c: Log is configured to *not* fsync() on all Append() calls
I20260810 00:50:08.063369 24842 tablet_bootstrap.cc:492] T ca4e72d29df945d28c32a34ab50cade0 P 18d6502671b14f558df3565d1e54169e: Bootstrap starting.
I20260810 00:50:08.063745 24840 tablet_bootstrap.cc:492] T ca4e72d29df945d28c32a34ab50cade0 P d9cccd0afb46475c9ef17a7001a15c3c: No bootstrap required, opened a new log
I20260810 00:50:08.064193 24840 ts_tablet_manager.cc:1397] T ca4e72d29df945d28c32a34ab50cade0 P d9cccd0afb46475c9ef17a7001a15c3c: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260810 00:50:08.066406 24842 tablet_bootstrap.cc:654] T ca4e72d29df945d28c32a34ab50cade0 P 18d6502671b14f558df3565d1e54169e: Neither blocks nor log segments found. Creating new log.
I20260810 00:50:08.067647 24846 tablet_bootstrap.cc:492] T ca4e72d29df945d28c32a34ab50cade0 P b61978fb2e8841cdb64c820d6dc0bb2b: Bootstrap starting.
I20260810 00:50:08.067677 24842 log.cc:824] T ca4e72d29df945d28c32a34ab50cade0 P 18d6502671b14f558df3565d1e54169e: Log is configured to *not* fsync() on all Append() calls
I20260810 00:50:08.070636 24842 tablet_bootstrap.cc:492] T ca4e72d29df945d28c32a34ab50cade0 P 18d6502671b14f558df3565d1e54169e: No bootstrap required, opened a new log
I20260810 00:50:08.070924 24842 ts_tablet_manager.cc:1397] T ca4e72d29df945d28c32a34ab50cade0 P 18d6502671b14f558df3565d1e54169e: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.002s
I20260810 00:50:08.071113 24846 tablet_bootstrap.cc:654] T ca4e72d29df945d28c32a34ab50cade0 P b61978fb2e8841cdb64c820d6dc0bb2b: Neither blocks nor log segments found. Creating new log.
I20260810 00:50:08.072067 24846 log.cc:824] T ca4e72d29df945d28c32a34ab50cade0 P b61978fb2e8841cdb64c820d6dc0bb2b: Log is configured to *not* fsync() on all Append() calls
I20260810 00:50:08.074473 24846 tablet_bootstrap.cc:492] T ca4e72d29df945d28c32a34ab50cade0 P b61978fb2e8841cdb64c820d6dc0bb2b: No bootstrap required, opened a new log
I20260810 00:50:08.074784 24846 ts_tablet_manager.cc:1397] T ca4e72d29df945d28c32a34ab50cade0 P b61978fb2e8841cdb64c820d6dc0bb2b: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260810 00:50:08.075937 24840 raft_consensus.cc:348] T ca4e72d29df945d28c32a34ab50cade0 P d9cccd0afb46475c9ef17a7001a15c3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cccd0afb46475c9ef17a7001a15c3c" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39925 } } peers { permanent_uuid: "b61978fb2e8841cdb64c820d6dc0bb2b" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 45219 } } peers { permanent_uuid: "18d6502671b14f558df3565d1e54169e" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 41673 } }
I20260810 00:50:08.076399 24840 raft_consensus.cc:374] T ca4e72d29df945d28c32a34ab50cade0 P d9cccd0afb46475c9ef17a7001a15c3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:50:08.076499 24840 raft_consensus.cc:729] T ca4e72d29df945d28c32a34ab50cade0 P d9cccd0afb46475c9ef17a7001a15c3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9cccd0afb46475c9ef17a7001a15c3c, State: Initialized, Role: FOLLOWER
I20260810 00:50:08.077117 24840 consensus_queue.cc:260] T ca4e72d29df945d28c32a34ab50cade0 P d9cccd0afb46475c9ef17a7001a15c3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cccd0afb46475c9ef17a7001a15c3c" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39925 } } peers { permanent_uuid: "b61978fb2e8841cdb64c820d6dc0bb2b" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 45219 } } peers { permanent_uuid: "18d6502671b14f558df3565d1e54169e" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 41673 } }
I20260810 00:50:08.078233 24809 heartbeater.cc:503] Master 127.28.6.62:35357 was elected leader, sending a full tablet report...
I20260810 00:50:08.078807 24840 ts_tablet_manager.cc:1428] T ca4e72d29df945d28c32a34ab50cade0 P d9cccd0afb46475c9ef17a7001a15c3c: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260810 00:50:08.081264 24842 raft_consensus.cc:348] T ca4e72d29df945d28c32a34ab50cade0 P 18d6502671b14f558df3565d1e54169e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cccd0afb46475c9ef17a7001a15c3c" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39925 } } peers { permanent_uuid: "b61978fb2e8841cdb64c820d6dc0bb2b" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 45219 } } peers { permanent_uuid: "18d6502671b14f558df3565d1e54169e" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 41673 } }
I20260810 00:50:08.081712 24842 raft_consensus.cc:374] T ca4e72d29df945d28c32a34ab50cade0 P 18d6502671b14f558df3565d1e54169e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:50:08.081842 24842 raft_consensus.cc:729] T ca4e72d29df945d28c32a34ab50cade0 P 18d6502671b14f558df3565d1e54169e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18d6502671b14f558df3565d1e54169e, State: Initialized, Role: FOLLOWER
I20260810 00:50:08.082461 24842 consensus_queue.cc:260] T ca4e72d29df945d28c32a34ab50cade0 P 18d6502671b14f558df3565d1e54169e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cccd0afb46475c9ef17a7001a15c3c" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39925 } } peers { permanent_uuid: "b61978fb2e8841cdb64c820d6dc0bb2b" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 45219 } } peers { permanent_uuid: "18d6502671b14f558df3565d1e54169e" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 41673 } }
I20260810 00:50:08.084291 24842 ts_tablet_manager.cc:1428] T ca4e72d29df945d28c32a34ab50cade0 P 18d6502671b14f558df3565d1e54169e: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260810 00:50:08.084050 24846 raft_consensus.cc:348] T ca4e72d29df945d28c32a34ab50cade0 P b61978fb2e8841cdb64c820d6dc0bb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cccd0afb46475c9ef17a7001a15c3c" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39925 } } peers { permanent_uuid: "b61978fb2e8841cdb64c820d6dc0bb2b" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 45219 } } peers { permanent_uuid: "18d6502671b14f558df3565d1e54169e" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 41673 } }
I20260810 00:50:08.084537 24846 raft_consensus.cc:374] T ca4e72d29df945d28c32a34ab50cade0 P b61978fb2e8841cdb64c820d6dc0bb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:50:08.084685 24846 raft_consensus.cc:729] T ca4e72d29df945d28c32a34ab50cade0 P b61978fb2e8841cdb64c820d6dc0bb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b61978fb2e8841cdb64c820d6dc0bb2b, State: Initialized, Role: FOLLOWER
I20260810 00:50:08.085302 24846 consensus_queue.cc:260] T ca4e72d29df945d28c32a34ab50cade0 P b61978fb2e8841cdb64c820d6dc0bb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cccd0afb46475c9ef17a7001a15c3c" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39925 } } peers { permanent_uuid: "b61978fb2e8841cdb64c820d6dc0bb2b" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 45219 } } peers { permanent_uuid: "18d6502671b14f558df3565d1e54169e" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 41673 } }
I20260810 00:50:08.086335 24590 heartbeater.cc:503] Master 127.28.6.62:35357 was elected leader, sending a full tablet report...
I20260810 00:50:08.086972 24846 ts_tablet_manager.cc:1428] T ca4e72d29df945d28c32a34ab50cade0 P b61978fb2e8841cdb64c820d6dc0bb2b: Time spent starting tablet: real 0.012s user 0.004s sys 0.010s
W20260810 00:50:08.141177 24378 tablet.cc:2369] T ca4e72d29df945d28c32a34ab50cade0 P 18d6502671b14f558df3565d1e54169e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:50:08.155714 24855 raft_consensus.cc:482] T ca4e72d29df945d28c32a34ab50cade0 P b61978fb2e8841cdb64c820d6dc0bb2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:50:08.156021 24855 raft_consensus.cc:504] T ca4e72d29df945d28c32a34ab50cade0 P b61978fb2e8841cdb64c820d6dc0bb2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cccd0afb46475c9ef17a7001a15c3c" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39925 } } peers { permanent_uuid: "b61978fb2e8841cdb64c820d6dc0bb2b" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 45219 } } peers { permanent_uuid: "18d6502671b14f558df3565d1e54169e" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 41673 } }
I20260810 00:50:08.157832 24855 leader_election.cc:288] T ca4e72d29df945d28c32a34ab50cade0 P b61978fb2e8841cdb64c820d6dc0bb2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9cccd0afb46475c9ef17a7001a15c3c (127.28.6.3:39925), 18d6502671b14f558df3565d1e54169e (127.28.6.1:41673)
I20260810 00:50:08.165516 24837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:08.158445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:52907 (local address 127.28.6.1:41673)
0810 00:50:08.158703 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:08.158713 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:08.159532 (+ 819us) server_negotiation.cc:407] Connection header received
0810 00:50:08.159805 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:08.159811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:08.159883 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:08.159979 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:50:08.160455 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:08.161344 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:08.163709 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:08.164512 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:08.164681 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:08.164799 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:08.165130 (+ 331us) server_negotiation.cc:299] Negotiation successful
0810 00:50:08.165250 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 00:50:08.165521 24830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:08.158141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:35601 (local address 127.28.6.3:39925)
0810 00:50:08.158387 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:08.158396 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:08.159761 (+ 1365us) server_negotiation.cc:407] Connection header received
0810 00:50:08.160071 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:08.160080 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:08.160168 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:08.160234 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:50:08.161101 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:08.161924 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:08.163795 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:08.164555 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:08.164705 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:08.164833 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:08.165134 (+ 301us) server_negotiation.cc:299] Negotiation successful
0810 00:50:08.165262 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 00:50:08.165705 24857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:08.158151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:41673 (local address 127.28.6.2:52907)
0810 00:50:08.159415 (+ 1264us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:08.159438 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:08.159653 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:08.159997 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:08.160002 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:08.160027 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:50:08.160315 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:08.160324 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:08.161490 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:08.161498 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:08.163544 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:08.163562 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:08.164763 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:08.164773 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:08.164950 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:08.164982 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 00:50:08.165090 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 00:50:08.165217 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1088,"mutex_wait_us":24,"thread_start_us":1125,"threads_started":1}
I20260810 00:50:08.165827 24856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:08.158190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:39925 (local address 127.28.6.2:35601)
0810 00:50:08.159650 (+ 1460us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:08.159670 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:08.159891 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:08.160445 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:08.160452 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:08.160482 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:50:08.160925 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:08.160941 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:08.162064 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:08.162070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:08.163607 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:08.163620 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:08.164791 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:08.164797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:08.164981 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:08.165004 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 00:50:08.165107 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 00:50:08.165226 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1318,"mutex_wait_us":4,"thread_start_us":536,"threads_started":1}
I20260810 00:50:08.166831 24736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca4e72d29df945d28c32a34ab50cade0" candidate_uuid: "b61978fb2e8841cdb64c820d6dc0bb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9cccd0afb46475c9ef17a7001a15c3c" is_pre_election: true
I20260810 00:50:08.167398 24736 raft_consensus.cc:2454] T ca4e72d29df945d28c32a34ab50cade0 P d9cccd0afb46475c9ef17a7001a15c3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b61978fb2e8841cdb64c820d6dc0bb2b in term 0.
I20260810 00:50:08.167397 24306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca4e72d29df945d28c32a34ab50cade0" candidate_uuid: "b61978fb2e8841cdb64c820d6dc0bb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18d6502671b14f558df3565d1e54169e" is_pre_election: true
I20260810 00:50:08.167933 24306 raft_consensus.cc:2454] T ca4e72d29df945d28c32a34ab50cade0 P 18d6502671b14f558df3565d1e54169e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b61978fb2e8841cdb64c820d6dc0bb2b in term 0.
I20260810 00:50:08.168099 24417 leader_election.cc:302] T ca4e72d29df945d28c32a34ab50cade0 P b61978fb2e8841cdb64c820d6dc0bb2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b61978fb2e8841cdb64c820d6dc0bb2b, d9cccd0afb46475c9ef17a7001a15c3c; no voters:
I20260810 00:50:08.168515 24855 raft_consensus.cc:2781] T ca4e72d29df945d28c32a34ab50cade0 P b61978fb2e8841cdb64c820d6dc0bb2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 00:50:08.168665 24855 raft_consensus.cc:482] T ca4e72d29df945d28c32a34ab50cade0 P b61978fb2e8841cdb64c820d6dc0bb2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 00:50:08.168780 24855 raft_consensus.cc:3037] T ca4e72d29df945d28c32a34ab50cade0 P b61978fb2e8841cdb64c820d6dc0bb2b [term 0 FOLLOWER]: Advancing to term 1
W20260810 00:50:08.170392 24591 tablet.cc:2369] T ca4e72d29df945d28c32a34ab50cade0 P b61978fb2e8841cdb64c820d6dc0bb2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:50:08.171108 24855 raft_consensus.cc:504] T ca4e72d29df945d28c32a34ab50cade0 P b61978fb2e8841cdb64c820d6dc0bb2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cccd0afb46475c9ef17a7001a15c3c" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39925 } } peers { permanent_uuid: "b61978fb2e8841cdb64c820d6dc0bb2b" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 45219 } } peers { permanent_uuid: "18d6502671b14f558df3565d1e54169e" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 41673 } }
I20260810 00:50:08.171707 24855 leader_election.cc:288] T ca4e72d29df945d28c32a34ab50cade0 P b61978fb2e8841cdb64c820d6dc0bb2b [CANDIDATE]: Term 1 election: Requested vote from peers d9cccd0afb46475c9ef17a7001a15c3c (127.28.6.3:39925), 18d6502671b14f558df3565d1e54169e (127.28.6.1:41673)
I20260810 00:50:08.172137 24736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca4e72d29df945d28c32a34ab50cade0" candidate_uuid: "b61978fb2e8841cdb64c820d6dc0bb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9cccd0afb46475c9ef17a7001a15c3c"
I20260810 00:50:08.172242 24306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca4e72d29df945d28c32a34ab50cade0" candidate_uuid: "b61978fb2e8841cdb64c820d6dc0bb2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18d6502671b14f558df3565d1e54169e"
I20260810 00:50:08.172351 24736 raft_consensus.cc:3037] T ca4e72d29df945d28c32a34ab50cade0 P d9cccd0afb46475c9ef17a7001a15c3c [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:50:08.172472 24306 raft_consensus.cc:3037] T ca4e72d29df945d28c32a34ab50cade0 P 18d6502671b14f558df3565d1e54169e [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:50:08.174605 24736 raft_consensus.cc:2454] T ca4e72d29df945d28c32a34ab50cade0 P d9cccd0afb46475c9ef17a7001a15c3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b61978fb2e8841cdb64c820d6dc0bb2b in term 1.
I20260810 00:50:08.174667 24306 raft_consensus.cc:2454] T ca4e72d29df945d28c32a34ab50cade0 P 18d6502671b14f558df3565d1e54169e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b61978fb2e8841cdb64c820d6dc0bb2b in term 1.
I20260810 00:50:08.175033 24417 leader_election.cc:302] T ca4e72d29df945d28c32a34ab50cade0 P b61978fb2e8841cdb64c820d6dc0bb2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b61978fb2e8841cdb64c820d6dc0bb2b, d9cccd0afb46475c9ef17a7001a15c3c; no voters:
I20260810 00:50:08.175343 24855 raft_consensus.cc:2781] T ca4e72d29df945d28c32a34ab50cade0 P b61978fb2e8841cdb64c820d6dc0bb2b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:50:08.175769 24855 raft_consensus.cc:686] T ca4e72d29df945d28c32a34ab50cade0 P b61978fb2e8841cdb64c820d6dc0bb2b [term 1 LEADER]: Becoming Leader. State: Replica: b61978fb2e8841cdb64c820d6dc0bb2b, State: Running, Role: LEADER
I20260810 00:50:08.176290 24855 consensus_queue.cc:237] T ca4e72d29df945d28c32a34ab50cade0 P b61978fb2e8841cdb64c820d6dc0bb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cccd0afb46475c9ef17a7001a15c3c" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39925 } } peers { permanent_uuid: "b61978fb2e8841cdb64c820d6dc0bb2b" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 45219 } } peers { permanent_uuid: "18d6502671b14f558df3565d1e54169e" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 41673 } }
I20260810 00:50:08.180755 24094 catalog_manager.cc:5495] T ca4e72d29df945d28c32a34ab50cade0 P b61978fb2e8841cdb64c820d6dc0bb2b reported cstate change: term changed from 0 to 1, leader changed from <none> to b61978fb2e8841cdb64c820d6dc0bb2b (127.28.6.2). New cstate: current_term: 1 leader_uuid: "b61978fb2e8841cdb64c820d6dc0bb2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cccd0afb46475c9ef17a7001a15c3c" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b61978fb2e8841cdb64c820d6dc0bb2b" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 45219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18d6502671b14f558df3565d1e54169e" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 41673 } health_report { overall_health: UNKNOWN } } }
W20260810 00:50:08.203753 24810 tablet.cc:2369] T ca4e72d29df945d28c32a34ab50cade0 P d9cccd0afb46475c9ef17a7001a15c3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:50:08.220765 24839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:08.215674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42872 (local address 127.28.6.2:45219)
0810 00:50:08.215987 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:08.215992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:08.216009 (+ 17us) server_negotiation.cc:407] Connection header received
0810 00:50:08.216095 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:08.216098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:08.216178 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:08.216301 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:50:08.216671 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:08.217392 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:08.218701 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:08.219108 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:08.219202 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:08.219676 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:50:08.220120 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:50:08.220259 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:08.220378 (+ 119us) server_negotiation.cc:299] Negotiation successful
0810 00:50:08.220446 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196}
I20260810 00:50:08.228467 24736 raft_consensus.cc:1261] T ca4e72d29df945d28c32a34ab50cade0 P d9cccd0afb46475c9ef17a7001a15c3c [term 1 FOLLOWER]: Refusing update from remote peer b61978fb2e8841cdb64c820d6dc0bb2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:50:08.228487 24306 raft_consensus.cc:1261] T ca4e72d29df945d28c32a34ab50cade0 P 18d6502671b14f558df3565d1e54169e [term 1 FOLLOWER]: Refusing update from remote peer b61978fb2e8841cdb64c820d6dc0bb2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:50:08.229413 24858 consensus_queue.cc:1035] T ca4e72d29df945d28c32a34ab50cade0 P b61978fb2e8841cdb64c820d6dc0bb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9cccd0afb46475c9ef17a7001a15c3c" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:50:08.229736 24855 consensus_queue.cc:1035] T ca4e72d29df945d28c32a34ab50cade0 P b61978fb2e8841cdb64c820d6dc0bb2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "18d6502671b14f558df3565d1e54169e" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 41673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:50:08.255084 24093 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:50:08.255492 24093 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:50:08.260409 24093 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e037c4be8b814040847febe7761ec66c: Sending DeleteTablet for 3 replicas of tablet ca4e72d29df945d28c32a34ab50cade0
I20260810 00:50:08.261703 24705 tablet_service.cc:1514] Processing DeleteTablet for tablet ca4e72d29df945d28c32a34ab50cade0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:50:08 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:53164
I20260810 00:50:08.261775 24487 tablet_service.cc:1514] Processing DeleteTablet for tablet ca4e72d29df945d28c32a34ab50cade0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:50:08 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:42858
I20260810 00:50:08.262384 24278 tablet_service.cc:1514] Processing DeleteTablet for tablet ca4e72d29df945d28c32a34ab50cade0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:50:08 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:39078
I20260810 00:50:08.262634 24879 tablet_replica.cc:330] T ca4e72d29df945d28c32a34ab50cade0 P b61978fb2e8841cdb64c820d6dc0bb2b: stopping tablet replica
I20260810 00:50:08.263029 24879 raft_consensus.cc:2229] T ca4e72d29df945d28c32a34ab50cade0 P b61978fb2e8841cdb64c820d6dc0bb2b [term 1 LEADER]: Raft consensus shutting down.
I20260810 00:50:08.263734 24879 raft_consensus.cc:2258] T ca4e72d29df945d28c32a34ab50cade0 P b61978fb2e8841cdb64c820d6dc0bb2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:50:08.265710 24879 ts_tablet_manager.cc:1905] T ca4e72d29df945d28c32a34ab50cade0 P b61978fb2e8841cdb64c820d6dc0bb2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:50:08.267354 24880 tablet_replica.cc:330] T ca4e72d29df945d28c32a34ab50cade0 P d9cccd0afb46475c9ef17a7001a15c3c: stopping tablet replica
I20260810 00:50:08.268319 24882 tablet_replica.cc:330] T ca4e72d29df945d28c32a34ab50cade0 P 18d6502671b14f558df3565d1e54169e: stopping tablet replica
I20260810 00:50:08.268756 24882 raft_consensus.cc:2229] T ca4e72d29df945d28c32a34ab50cade0 P 18d6502671b14f558df3565d1e54169e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:50:08.269224 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 24182
I20260810 00:50:08.269294 24882 raft_consensus.cc:2258] T ca4e72d29df945d28c32a34ab50cade0 P 18d6502671b14f558df3565d1e54169e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:50:08.273281 24879 ts_tablet_manager.cc:1918] T ca4e72d29df945d28c32a34ab50cade0 P b61978fb2e8841cdb64c820d6dc0bb2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:50:08.273509 24879 log.cc:1196] T ca4e72d29df945d28c32a34ab50cade0 P b61978fb2e8841cdb64c820d6dc0bb2b: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-1/wal/wals/ca4e72d29df945d28c32a34ab50cade0
I20260810 00:50:08.274673 24879 ts_tablet_manager.cc:1939] T ca4e72d29df945d28c32a34ab50cade0 P b61978fb2e8841cdb64c820d6dc0bb2b: Deleting consensus metadata
I20260810 00:50:08.275200 24882 ts_tablet_manager.cc:1905] T ca4e72d29df945d28c32a34ab50cade0 P 18d6502671b14f558df3565d1e54169e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:50:08.276474 24075 catalog_manager.cc:4841] TS b61978fb2e8841cdb64c820d6dc0bb2b (127.28.6.2:45219): tablet ca4e72d29df945d28c32a34ab50cade0 (table test-table [id=ca93f315a9d243ea999240114a5d290c]) successfully deleted
I20260810 00:50:08.279261 24880 raft_consensus.cc:2229] T ca4e72d29df945d28c32a34ab50cade0 P d9cccd0afb46475c9ef17a7001a15c3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:50:08.280236 24880 raft_consensus.cc:2258] T ca4e72d29df945d28c32a34ab50cade0 P d9cccd0afb46475c9ef17a7001a15c3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:50:08.280597 24882 ts_tablet_manager.cc:1918] T ca4e72d29df945d28c32a34ab50cade0 P 18d6502671b14f558df3565d1e54169e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:50:08.280833 24882 log.cc:1196] T ca4e72d29df945d28c32a34ab50cade0 P 18d6502671b14f558df3565d1e54169e: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-0/wal/wals/ca4e72d29df945d28c32a34ab50cade0
I20260810 00:50:08.281459 24882 ts_tablet_manager.cc:1939] T ca4e72d29df945d28c32a34ab50cade0 P 18d6502671b14f558df3565d1e54169e: Deleting consensus metadata
I20260810 00:50:08.283496 24069 catalog_manager.cc:4841] TS 18d6502671b14f558df3565d1e54169e (127.28.6.1:41673): tablet ca4e72d29df945d28c32a34ab50cade0 (table test-table [id=ca93f315a9d243ea999240114a5d290c]) successfully deleted
I20260810 00:50:08.285586 24880 ts_tablet_manager.cc:1905] T ca4e72d29df945d28c32a34ab50cade0 P d9cccd0afb46475c9ef17a7001a15c3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:50:08.290680 24880 ts_tablet_manager.cc:1918] T ca4e72d29df945d28c32a34ab50cade0 P d9cccd0afb46475c9ef17a7001a15c3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:50:08.290848 24880 log.cc:1196] T ca4e72d29df945d28c32a34ab50cade0 P d9cccd0afb46475c9ef17a7001a15c3c: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786322687226232-28696-0/minicluster-data/ts-2/wal/wals/ca4e72d29df945d28c32a34ab50cade0
I20260810 00:50:08.291447 24880 ts_tablet_manager.cc:1939] T ca4e72d29df945d28c32a34ab50cade0 P d9cccd0afb46475c9ef17a7001a15c3c: Deleting consensus metadata
Aug 10 00:50:08 dist-test-slave-1nl6 krb5kdc[24021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.1@KRBTEST.COM
I20260810 00:50:08.292922 24075 catalog_manager.cc:4841] TS d9cccd0afb46475c9ef17a7001a15c3c (127.28.6.3:39925): tablet ca4e72d29df945d28c32a34ab50cade0 (table test-table [id=ca93f315a9d243ea999240114a5d290c]) successfully deleted
I20260810 00:50:08.299274 24837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:08.271760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39088 (local address 127.28.6.1:41673)
0810 00:50:08.271932 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:08.271939 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:08.280606 (+ 8667us) server_negotiation.cc:407] Connection header received
0810 00:50:08.280838 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:08.280843 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:08.280929 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:08.281059 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:08.283937 (+ 2878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:08.284976 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:08.286842 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:08.287483 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:08.287621 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:08.293215 (+ 5594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:08.293225 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:08.293240 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:50:08.293270 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:08.296228 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:08.297071 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:08.297078 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:08.297080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:08.297196 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:08.297635 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:08.297639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:08.297641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:08.298010 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:08.298226 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:08.298592 (+ 366us) server_negotiation.cc:299] Negotiation successful
0810 00:50:08.298838 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 00:50:08.299930 24366 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:50:08.450344 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 24182
I20260810 00:50:08.486073 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 24384
Aug 10 00:50:08 dist-test-slave-1nl6 krb5kdc[24021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.2@KRBTEST.COM
I20260810 00:50:08.506214 24839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:08.486933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42880 (local address 127.28.6.2:45219)
0810 00:50:08.487120 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:08.487125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:08.487209 (+ 84us) server_negotiation.cc:407] Connection header received
0810 00:50:08.487395 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:08.487398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:08.487468 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:08.487549 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:08.489313 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:08.490536 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:08.492685 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:08.493473 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:08.493631 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:08.498821 (+ 5190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:08.498841 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:08.498873 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:50:08.499177 (+ 304us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:08.503417 (+ 4240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:08.504075 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:08.504085 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:08.504089 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:08.504260 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:08.504606 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:08.504615 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:08.504619 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:08.505076 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:08.505418 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:08.505597 (+ 179us) server_negotiation.cc:299] Negotiation successful
0810 00:50:08.505824 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 00:50:08.507140 24579 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:50:08.618798 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 24384
I20260810 00:50:08.646194 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 24598
Aug 10 00:50:08 dist-test-slave-1nl6 krb5kdc[24021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.3@KRBTEST.COM
I20260810 00:50:08.661273 24830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:08.647007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53176 (local address 127.28.6.3:39925)
0810 00:50:08.647207 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:08.647212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:08.647231 (+ 19us) server_negotiation.cc:407] Connection header received
0810 00:50:08.647374 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:08.647378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:08.647445 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:08.647529 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:08.648848 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:08.649544 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:08.650790 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:08.651348 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:08.651443 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:08.656146 (+ 4703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:08.656158 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:08.656175 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:50:08.656211 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:08.658727 (+ 2516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:08.659416 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:08.659424 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:08.659426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:08.659563 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:08.659944 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:08.659947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:08.659949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:08.660290 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:08.660508 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:08.660753 (+ 245us) server_negotiation.cc:299] Negotiation successful
0810 00:50:08.660904 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 00:50:08.661931 24799 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:50:08.749415 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 24598
I20260810 00:50:08.772718 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 24041
I20260810 00:50:08.773779 24135 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:50:08.880789 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 24041
2026-08-10T00:50:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4672 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-10T00:50:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:50:08Z Disabled control of system clock
I20260810 00:50:08.973362 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:37253
--webserver_interface=127.28.6.62
--webserver_port=0
--builtin_ntp_servers=127.28.6.20:39287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:37253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 00:50:09.308566 24897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:50:09.308833 24897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:50:09.308967 24897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:50:09.316179 24897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:50:09.316304 24897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:50:09.316344 24897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:50:09.316414 24897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:50:09.316483 24897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:50:09.325304 24897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:39287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:37253
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:37253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:50:09.326503 24897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:50:09.328266 24897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:50:09.339874 24906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:09.340144 24910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:09.340139 24905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:50:09.340978 24897 server_base.cc:1034] running on GCE node
I20260810 00:50:09.341734 24897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:50:09.343142 24897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:50:09.344384 24897 hybrid_clock.cc:648] HybridClock initialized: now 1786323009344340 us; error 39 us; skew 500 ppm
I20260810 00:50:09.344729 24897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 00:50:09.347047 24897 webserver.cc:460] Webserver started at http://127.28.6.62:43841/ using document root <none> and password file <none>
I20260810 00:50:09.347725 24897 fs_manager.cc:359] Metadata directory not provided
I20260810 00:50:09.347858 24897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:50:09.348182 24897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:50:09.351109 24897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "ba68d1b46992431ebf99b05cf2daede6"
format_stamp: "Formatted at 2026-08-10 00:50:09 on dist-test-slave-1nl6"
server_key: "a199f2fc7b95d3a10656a209a53e289c"
server_key_iv: "9b686b3679fdc23ddfd96493dd6904d1"
server_key_version: "encryptionkey@0"
I20260810 00:50:09.351939 24897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "ba68d1b46992431ebf99b05cf2daede6"
format_stamp: "Formatted at 2026-08-10 00:50:09 on dist-test-slave-1nl6"
server_key: "a199f2fc7b95d3a10656a209a53e289c"
server_key_iv: "9b686b3679fdc23ddfd96493dd6904d1"
server_key_version: "encryptionkey@0"
I20260810 00:50:09.356926 24897 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 00:50:09.360213 24917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:09.361462 24897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 00:50:09.361658 24897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "ba68d1b46992431ebf99b05cf2daede6"
format_stamp: "Formatted at 2026-08-10 00:50:09 on dist-test-slave-1nl6"
server_key: "a199f2fc7b95d3a10656a209a53e289c"
server_key_iv: "9b686b3679fdc23ddfd96493dd6904d1"
server_key_version: "encryptionkey@0"
I20260810 00:50:09.361871 24897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:50:09.385144 24897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:50:09.386006 24897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:50:09.386305 24897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:50:09.405122 24897 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:37253
I20260810 00:50:09.405161 24993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:37253 every 8 connection(s)
I20260810 00:50:09.406436 24897 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:50:09.407346 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 24897
I20260810 00:50:09.407550 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:50:09.408129 28696 external_mini_cluster.cc:1428] Setting key 8bb3d8d651bff98b2c7c88238f1402b6
I20260810 00:50:09.411281 24997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:50:09.421365 25000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:09.410094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34144 (local address 127.28.6.62:37253)
0810 00:50:09.411138 (+ 1044us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:09.411148 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:09.411190 (+ 42us) server_negotiation.cc:407] Connection header received
0810 00:50:09.412875 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:09.412896 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:09.413323 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:09.413751 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:09.414309 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:09.415658 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:09.417009 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:09.417658 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:09.417873 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:09.419178 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:09.419185 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:09.419198 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:09.419746 (+ 548us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:09.419826 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:09.419857 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:09.420001 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:09.420233 (+ 232us) server_negotiation.cc:299] Negotiation successful
0810 00:50:09.420522 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":360,"threads_started":1}
I20260810 00:50:09.422080 24997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba68d1b46992431ebf99b05cf2daede6: Bootstrap starting.
I20260810 00:50:09.425710 24997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba68d1b46992431ebf99b05cf2daede6: Neither blocks nor log segments found. Creating new log.
I20260810 00:50:09.426764 24997 log.cc:824] T 00000000000000000000000000000000 P ba68d1b46992431ebf99b05cf2daede6: Log is configured to *not* fsync() on all Append() calls
I20260810 00:50:09.429704 24997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba68d1b46992431ebf99b05cf2daede6: No bootstrap required, opened a new log
I20260810 00:50:09.437186 24997 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba68d1b46992431ebf99b05cf2daede6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba68d1b46992431ebf99b05cf2daede6" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 37253 } }
I20260810 00:50:09.437541 24997 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba68d1b46992431ebf99b05cf2daede6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:50:09.437652 24997 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba68d1b46992431ebf99b05cf2daede6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba68d1b46992431ebf99b05cf2daede6, State: Initialized, Role: FOLLOWER
I20260810 00:50:09.438182 24997 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba68d1b46992431ebf99b05cf2daede6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba68d1b46992431ebf99b05cf2daede6" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 37253 } }
I20260810 00:50:09.438426 24997 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba68d1b46992431ebf99b05cf2daede6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:50:09.438583 24997 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba68d1b46992431ebf99b05cf2daede6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:50:09.438766 24997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba68d1b46992431ebf99b05cf2daede6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:50:09.441022 24997 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba68d1b46992431ebf99b05cf2daede6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba68d1b46992431ebf99b05cf2daede6" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 37253 } }
I20260810 00:50:09.441591 24997 leader_election.cc:302] T 00000000000000000000000000000000 P ba68d1b46992431ebf99b05cf2daede6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba68d1b46992431ebf99b05cf2daede6; no voters:
I20260810 00:50:09.442281 24997 leader_election.cc:288] T 00000000000000000000000000000000 P ba68d1b46992431ebf99b05cf2daede6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:50:09.442452 25004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba68d1b46992431ebf99b05cf2daede6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:50:09.443259 25004 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba68d1b46992431ebf99b05cf2daede6 [term 1 LEADER]: Becoming Leader. State: Replica: ba68d1b46992431ebf99b05cf2daede6, State: Running, Role: LEADER
I20260810 00:50:09.443833 25004 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba68d1b46992431ebf99b05cf2daede6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba68d1b46992431ebf99b05cf2daede6" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 37253 } }
I20260810 00:50:09.444391 24997 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba68d1b46992431ebf99b05cf2daede6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:50:09.446519 25005 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba68d1b46992431ebf99b05cf2daede6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba68d1b46992431ebf99b05cf2daede6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba68d1b46992431ebf99b05cf2daede6" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 37253 } } }
I20260810 00:50:09.446520 25008 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba68d1b46992431ebf99b05cf2daede6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba68d1b46992431ebf99b05cf2daede6. Latest consensus state: current_term: 1 leader_uuid: "ba68d1b46992431ebf99b05cf2daede6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba68d1b46992431ebf99b05cf2daede6" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 37253 } } }
I20260810 00:50:09.447003 25005 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba68d1b46992431ebf99b05cf2daede6 [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:09.446991 25008 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba68d1b46992431ebf99b05cf2daede6 [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:09.448868 25011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:50:09.453306 25011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:50:09.462592 25011 catalog_manager.cc:1347] Generated new cluster ID: 43891c65a2d843da827f4d159eac6c40
I20260810 00:50:09.462695 25011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:50:09.476686 25011 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:50:09.478430 25011 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:50:09.487282 25011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba68d1b46992431ebf99b05cf2daede6: Generated new TSK 0
I20260810 00:50:09.488237 25011 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 00:50:09.496644 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.1:0
--local_ip_for_outbound_sockets=127.28.6.1
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:37253
--builtin_ntp_servers=127.28.6.20:39287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 00:50:09.837846 25030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:50:09.838100 25030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:50:09.838204 25030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:50:09.845652 25030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:50:09.845742 25030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:50:09.845839 25030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.1
I20260810 00:50:09.854497 25030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:39287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:37253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.1
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:50:09.855734 25030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:50:09.857676 25030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:50:09.870584 25042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:09.870656 25044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:09.870687 25039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:50:09.871644 25030 server_base.cc:1034] running on GCE node
I20260810 00:50:09.872224 25030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:50:09.873081 25030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:50:09.874282 25030 hybrid_clock.cc:648] HybridClock initialized: now 1786323009874241 us; error 33 us; skew 500 ppm
I20260810 00:50:09.874584 25030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 00:50:09.876829 25030 webserver.cc:460] Webserver started at http://127.28.6.1:34497/ using document root <none> and password file <none>
I20260810 00:50:09.877492 25030 fs_manager.cc:359] Metadata directory not provided
I20260810 00:50:09.877609 25030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:50:09.877930 25030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:50:09.880722 25030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-0/data/instance:
uuid: "e0b7d3c6bf5c401db17583908c4f5fe6"
format_stamp: "Formatted at 2026-08-10 00:50:09 on dist-test-slave-1nl6"
server_key: "ba74f113d52956dc151a2ccfc68b5d21"
server_key_iv: "4940805e33b50644f46a04329f39a542"
server_key_version: "encryptionkey@0"
I20260810 00:50:09.881493 25030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance:
uuid: "e0b7d3c6bf5c401db17583908c4f5fe6"
format_stamp: "Formatted at 2026-08-10 00:50:09 on dist-test-slave-1nl6"
server_key: "ba74f113d52956dc151a2ccfc68b5d21"
server_key_iv: "4940805e33b50644f46a04329f39a542"
server_key_version: "encryptionkey@0"
I20260810 00:50:09.886420 25030 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260810 00:50:09.889624 25054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:09.890842 25030 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 00:50:09.891047 25030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-0/wal
uuid: "e0b7d3c6bf5c401db17583908c4f5fe6"
format_stamp: "Formatted at 2026-08-10 00:50:09 on dist-test-slave-1nl6"
server_key: "ba74f113d52956dc151a2ccfc68b5d21"
server_key_iv: "4940805e33b50644f46a04329f39a542"
server_key_version: "encryptionkey@0"
I20260810 00:50:09.891242 25030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:50:09.937857 25030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:50:09.939332 25030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:50:09.940742 25030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:50:09.942051 25030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:50:09.944185 25030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:50:09.944316 25030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:09.944451 25030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:50:09.944535 25030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:09.983450 25030 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.1:40663
I20260810 00:50:09.983528 25230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.1:40663 every 8 connection(s)
I20260810 00:50:09.984776 25030 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
I20260810 00:50:09.986967 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 25030
I20260810 00:50:09.987157 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance
I20260810 00:50:09.987762 28696 external_mini_cluster.cc:1428] Setting key 905edb39ff037cf63f3006e5eca1770b
I20260810 00:50:09.990872 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.2:0
--local_ip_for_outbound_sockets=127.28.6.2
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:37253
--builtin_ntp_servers=127.28.6.20:39287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 00:50:09.998229 25234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:09.987583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:57801 (local address 127.28.6.62:37253)
0810 00:50:09.988039 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:09.988044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:09.989508 (+ 1464us) server_negotiation.cc:407] Connection header received
0810 00:50:09.991526 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:09.991531 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:09.991624 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:09.991743 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:09.992799 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:09.993511 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:09.995408 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:09.995844 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:09.995982 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:09.997360 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:09.997365 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:09.997368 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:09.997386 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:09.997426 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:09.997438 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:09.997538 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:09.997771 (+ 233us) server_negotiation.cc:299] Negotiation successful
0810 00:50:09.997871 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":276,"threads_started":1}
I20260810 00:50:09.999290 25236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:09.988271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:37253 (local address 127.28.6.1:57801)
0810 00:50:09.989359 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:09.989395 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:09.991272 (+ 1877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:09.991983 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:09.991991 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:09.992081 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:50:09.992625 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:09.992641 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:09.993674 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:09.993681 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:09.995255 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:09.995269 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:09.996087 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:09.996095 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:09.996328 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:09.997044 (+ 716us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 00:50:09.997046 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:50:09.997088 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 00:50:09.997158 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:50:09.997539 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:50:09.997554 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:50:09.997564 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 00:50:09.997763 (+ 199us) client_negotiation.cc:240] Negotiation successful
0810 00:50:09.998140 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":359,"threads_started":1}
I20260810 00:50:10.000869 25232 heartbeater.cc:347] Connected to a master server at 127.28.6.62:37253
I20260810 00:50:10.001163 25232 heartbeater.cc:464] Registering TS with master...
I20260810 00:50:10.001904 25232 heartbeater.cc:511] Master 127.28.6.62:37253 requested a full tablet report, sending...
I20260810 00:50:10.004065 24944 ts_manager.cc:194] Registered new tserver with Master: e0b7d3c6bf5c401db17583908c4f5fe6 (127.28.6.1:40663)
I20260810 00:50:10.005527 24944 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.6.1:57801
W20260810 00:50:10.339485 25237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:50:10.339802 25237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:50:10.339970 25237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:50:10.347330 25237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:50:10.347468 25237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:50:10.347630 25237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.2
I20260810 00:50:10.356684 25237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:39287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:37253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.2
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:50:10.357960 25237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:50:10.359819 25237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:50:10.373027 25244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:10.373086 25245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:10.373121 25247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:50:10.373919 25237 server_base.cc:1034] running on GCE node
I20260810 00:50:10.374457 25237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:50:10.375362 25237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:50:10.376571 25237 hybrid_clock.cc:648] HybridClock initialized: now 1786323010376524 us; error 42 us; skew 500 ppm
I20260810 00:50:10.376891 25237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 00:50:10.379202 25237 webserver.cc:460] Webserver started at http://127.28.6.2:33041/ using document root <none> and password file <none>
I20260810 00:50:10.379871 25237 fs_manager.cc:359] Metadata directory not provided
I20260810 00:50:10.380002 25237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:50:10.380330 25237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:50:10.383165 25237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-1/data/instance:
uuid: "7cd64ac068c5454fb4c64664c89b6489"
format_stamp: "Formatted at 2026-08-10 00:50:10 on dist-test-slave-1nl6"
server_key: "6941c67f05f896bd95b09d9e485de450"
server_key_iv: "92cb761c406a1ad45605faa7a5d7416e"
server_key_version: "encryptionkey@0"
I20260810 00:50:10.383970 25237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance:
uuid: "7cd64ac068c5454fb4c64664c89b6489"
format_stamp: "Formatted at 2026-08-10 00:50:10 on dist-test-slave-1nl6"
server_key: "6941c67f05f896bd95b09d9e485de450"
server_key_iv: "92cb761c406a1ad45605faa7a5d7416e"
server_key_version: "encryptionkey@0"
I20260810 00:50:10.388895 25237 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 00:50:10.392225 25255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:10.393437 25237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 00:50:10.393615 25237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-1/wal
uuid: "7cd64ac068c5454fb4c64664c89b6489"
format_stamp: "Formatted at 2026-08-10 00:50:10 on dist-test-slave-1nl6"
server_key: "6941c67f05f896bd95b09d9e485de450"
server_key_iv: "92cb761c406a1ad45605faa7a5d7416e"
server_key_version: "encryptionkey@0"
I20260810 00:50:10.393828 25237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:50:10.413393 25237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:50:10.414132 25237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:50:10.414433 25237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:50:10.415443 25237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:50:10.416893 25237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:50:10.417021 25237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:10.417148 25237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:50:10.417237 25237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:10.451243 25237 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.2:36555
I20260810 00:50:10.451344 25437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.2:36555 every 8 connection(s)
I20260810 00:50:10.452540 25237 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
I20260810 00:50:10.453462 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 25237
I20260810 00:50:10.453639 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance
I20260810 00:50:10.454164 28696 external_mini_cluster.cc:1428] Setting key 436bec552fd2bc97bf9ab7b46277ce7a
I20260810 00:50:10.457384 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.3:0
--local_ip_for_outbound_sockets=127.28.6.3
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:37253
--builtin_ntp_servers=127.28.6.20:39287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 00:50:10.467557 25234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:10.455186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:44313 (local address 127.28.6.62:37253)
0810 00:50:10.455366 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:10.455370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:10.457107 (+ 1737us) server_negotiation.cc:407] Connection header received
0810 00:50:10.459514 (+ 2407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:10.459522 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:10.459612 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:10.459725 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:10.460934 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:10.462091 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:10.464285 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:10.465017 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:10.465178 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:10.466601 (+ 1423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:10.466610 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:10.466617 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:10.466643 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:10.466688 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:10.466707 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:10.466803 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:10.467061 (+ 258us) server_negotiation.cc:299] Negotiation successful
0810 00:50:10.467168 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 00:50:10.468293 25440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:10.455792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:37253 (local address 127.28.6.2:44313)
0810 00:50:10.456933 (+ 1141us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:10.456984 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:10.459234 (+ 2250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:10.459938 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:10.459950 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:10.460051 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:50:10.460750 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:10.460773 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:10.462238 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:10.462246 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:10.464125 (+ 1879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:10.464142 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:10.465240 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:10.465247 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:10.465462 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:10.466249 (+ 787us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 00:50:10.466251 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:50:10.466287 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 00:50:10.466420 (+ 133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:50:10.466805 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:50:10.466825 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:50:10.466831 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 00:50:10.467022 (+ 191us) client_negotiation.cc:240] Negotiation successful
0810 00:50:10.467287 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":413,"threads_started":1}
I20260810 00:50:10.469702 25438 heartbeater.cc:347] Connected to a master server at 127.28.6.62:37253
I20260810 00:50:10.470031 25438 heartbeater.cc:464] Registering TS with master...
I20260810 00:50:10.470799 25438 heartbeater.cc:511] Master 127.28.6.62:37253 requested a full tablet report, sending...
I20260810 00:50:10.472313 24944 ts_manager.cc:194] Registered new tserver with Master: 7cd64ac068c5454fb4c64664c89b6489 (127.28.6.2:36555)
I20260810 00:50:10.473554 24944 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.6.2:44313
W20260810 00:50:10.802714 25441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:50:10.802982 25441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:50:10.803153 25441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:50:10.809994 25441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:50:10.810081 25441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:50:10.810284 25441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.3
I20260810 00:50:10.818836 25441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:39287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:37253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.3
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:50:10.819973 25441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:50:10.821691 25441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:50:10.834544 25453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:10.834630 25448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:10.834573 25449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:50:10.835770 25441 server_base.cc:1034] running on GCE node
I20260810 00:50:10.836371 25441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:50:10.837249 25441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:50:10.838480 25441 hybrid_clock.cc:648] HybridClock initialized: now 1786323010838435 us; error 56 us; skew 500 ppm
I20260810 00:50:10.838794 25441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 00:50:10.840927 25441 webserver.cc:460] Webserver started at http://127.28.6.3:45515/ using document root <none> and password file <none>
I20260810 00:50:10.841526 25441 fs_manager.cc:359] Metadata directory not provided
I20260810 00:50:10.841609 25441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:50:10.841856 25441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:50:10.844753 25441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-2/data/instance:
uuid: "195f200d5106466d98fb676fb6792d80"
format_stamp: "Formatted at 2026-08-10 00:50:10 on dist-test-slave-1nl6"
server_key: "f5ca559d1509725757d2fa0e87ea6a58"
server_key_iv: "ac0424deeafd881d8916765f85a834d5"
server_key_version: "encryptionkey@0"
I20260810 00:50:10.845464 25441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance:
uuid: "195f200d5106466d98fb676fb6792d80"
format_stamp: "Formatted at 2026-08-10 00:50:10 on dist-test-slave-1nl6"
server_key: "f5ca559d1509725757d2fa0e87ea6a58"
server_key_iv: "ac0424deeafd881d8916765f85a834d5"
server_key_version: "encryptionkey@0"
I20260810 00:50:10.850277 25441 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260810 00:50:10.853549 25461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:10.854871 25441 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 00:50:10.855140 25441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-2/wal
uuid: "195f200d5106466d98fb676fb6792d80"
format_stamp: "Formatted at 2026-08-10 00:50:10 on dist-test-slave-1nl6"
server_key: "f5ca559d1509725757d2fa0e87ea6a58"
server_key_iv: "ac0424deeafd881d8916765f85a834d5"
server_key_version: "encryptionkey@0"
I20260810 00:50:10.855502 25441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:50:10.878821 25441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:50:10.879645 25441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:50:10.879932 25441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:50:10.880849 25441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:50:10.882289 25441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:50:10.882421 25441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:10.882545 25441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:50:10.882637 25441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:10.914827 25441 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.3:39733
I20260810 00:50:10.914937 25636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.3:39733 every 8 connection(s)
I20260810 00:50:10.916095 25441 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
I20260810 00:50:10.920379 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 25441
I20260810 00:50:10.920584 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance
I20260810 00:50:10.921219 28696 external_mini_cluster.cc:1428] Setting key dfe07fb73f23587d7df8d024adc04072
I20260810 00:50:10.928913 25234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:10.918652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:52799 (local address 127.28.6.62:37253)
0810 00:50:10.918841 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:10.918847 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:10.920435 (+ 1588us) server_negotiation.cc:407] Connection header received
0810 00:50:10.922750 (+ 2315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:10.922754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:10.922818 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:10.922997 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:10.924178 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:10.924957 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:10.926446 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:10.926826 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:10.926968 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:10.928258 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:10.928263 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:10.928266 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:10.928281 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:10.928307 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:10.928318 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:10.928384 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:10.928576 (+ 192us) server_negotiation.cc:299] Negotiation successful
0810 00:50:10.928651 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 00:50:10.929787 25640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:10.919215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:37253 (local address 127.28.6.3:52799)
0810 00:50:10.920247 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:10.920300 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:10.922436 (+ 2136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:10.923243 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:10.923255 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:10.923383 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:50:10.923996 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:10.924025 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:10.925100 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:10.925103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:10.926313 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:10.926326 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:10.927082 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:10.927087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:10.927275 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:10.927987 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 00:50:10.927989 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:50:10.928025 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 00:50:10.928099 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:50:10.928402 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:50:10.928410 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:50:10.928416 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 00:50:10.928546 (+ 130us) client_negotiation.cc:240] Negotiation successful
0810 00:50:10.928826 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":286,"threads_started":1}
I20260810 00:50:10.931066 25638 heartbeater.cc:347] Connected to a master server at 127.28.6.62:37253
I20260810 00:50:10.931350 25638 heartbeater.cc:464] Registering TS with master...
I20260810 00:50:10.932034 25638 heartbeater.cc:511] Master 127.28.6.62:37253 requested a full tablet report, sending...
I20260810 00:50:10.933398 24944 ts_manager.cc:194] Registered new tserver with Master: 195f200d5106466d98fb676fb6792d80 (127.28.6.3:39733)
I20260810 00:50:10.934168 24944 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.6.3:52799
I20260810 00:50:10.936491 28696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 00:50:10.946612 25234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:10.940513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34148 (local address 127.28.6.62:37253)
0810 00:50:10.940668 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:10.940672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:10.941175 (+ 503us) server_negotiation.cc:407] Connection header received
0810 00:50:10.941358 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:10.941364 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:10.941432 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:10.941554 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:10.942101 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:10.942743 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:10.944273 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:10.944760 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:10.944906 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:10.945975 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:10.945981 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:10.945986 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:10.946011 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:10.946039 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:10.946052 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:10.946159 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:10.946353 (+ 194us) server_negotiation.cc:299] Negotiation successful
0810 00:50:10.946402 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 00:50:10.950033 24944 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34148:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 00:50:10.956943 24944 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 00:50:10.991858 25655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:10.968742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52478 (local address 127.28.6.1:40663)
0810 00:50:10.981480 (+ 12738us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:10.981491 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:10.981528 (+ 37us) server_negotiation.cc:407] Connection header received
0810 00:50:10.981653 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:10.981661 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:10.981919 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:10.983901 (+ 1982us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:50:10.984202 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:10.985986 (+ 1784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:10.989279 (+ 3293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:10.990249 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:10.990414 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:10.990719 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:10.991316 (+ 597us) server_negotiation.cc:299] Negotiation successful
0810 00:50:10.991483 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12545,"thread_start_us":12407,"threads_started":1}
I20260810 00:50:10.991838 25654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:10.968541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:40663 (local address 127.0.0.1:52478)
0810 00:50:10.975447 (+ 6906us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:10.975500 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:10.975680 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:10.982366 (+ 6686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:10.982373 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:10.982402 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:50:10.982862 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:10.982879 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:10.986144 (+ 3265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:10.986149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:10.988901 (+ 2752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:10.988924 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:10.990639 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:10.990651 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:10.991050 (+ 399us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:10.991090 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 00:50:10.991263 (+ 173us) client_negotiation.cc:240] Negotiation successful
0810 00:50:10.991446 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6654,"mutex_wait_us":7,"thread_start_us":14488,"threads_started":1}
I20260810 00:50:10.993424 25660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:10.968632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:36555 (local address 127.0.0.1:46184)
0810 00:50:10.981958 (+ 13326us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:10.981981 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:10.982162 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:10.983559 (+ 1397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:10.983564 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:10.983587 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:50:10.983954 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:10.983967 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:10.986012 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:10.986017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:10.989209 (+ 3192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:10.989225 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:10.992629 (+ 3404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:10.992635 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:10.992815 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:10.992847 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 00:50:10.992944 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 00:50:10.993073 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13136,"mutex_wait_us":239,"thread_start_us":6419,"threads_started":1}
I20260810 00:50:10.993530 25657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:10.969175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46184 (local address 127.28.6.2:36555)
0810 00:50:10.980770 (+ 11595us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:10.980781 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:10.982181 (+ 1400us) server_negotiation.cc:407] Connection header received
0810 00:50:10.983104 (+ 923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:10.983113 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:10.983325 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:10.984189 (+ 864us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:50:10.984415 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:10.985853 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:10.990884 (+ 5031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:10.992378 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:10.992560 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:10.992789 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:10.993003 (+ 214us) server_negotiation.cc:299] Negotiation successful
0810 00:50:10.993168 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11417,"thread_start_us":11613,"threads_started":1}
I20260810 00:50:10.996172 25119 tablet_service.cc:1467] Processing CreateTablet for tablet 9878436561e94139ab55805b54912f38 (DEFAULT_TABLE table=test-table [id=2d6de86fa4534b90903ff241621aab6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:50:10.997196 25658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:10.976260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56260 (local address 127.28.6.3:39733)
0810 00:50:10.982827 (+ 6567us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:10.982838 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:10.983399 (+ 561us) server_negotiation.cc:407] Connection header received
0810 00:50:10.983708 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:10.983715 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:10.983911 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:10.984095 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:50:10.985031 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:10.986490 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:10.991809 (+ 5319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:10.993225 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:10.993330 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:10.993588 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:10.994326 (+ 738us) server_negotiation.cc:299] Negotiation successful
0810 00:50:10.994489 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6351,"thread_start_us":6970,"threads_started":1}
I20260810 00:50:10.996754 25324 tablet_service.cc:1467] Processing CreateTablet for tablet 9878436561e94139ab55805b54912f38 (DEFAULT_TABLE table=test-table [id=2d6de86fa4534b90903ff241621aab6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:50:10.997680 25119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9878436561e94139ab55805b54912f38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:50:10.997974 25653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:10.980657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:39733 (local address 127.0.0.1:56260)
0810 00:50:10.983295 (+ 2638us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:10.983315 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:10.983505 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:10.984455 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:10.984461 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:10.984488 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:50:10.984840 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:10.984852 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:10.986723 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:10.986728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:10.990701 (+ 3973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:10.990718 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:10.993937 (+ 3219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:10.993944 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:10.994106 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:10.994135 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 00:50:10.997439 (+ 3304us) client_negotiation.cc:240] Negotiation successful
0810 00:50:10.997617 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2505,"mutex_wait_us":6,"thread_start_us":940,"threads_started":1}
I20260810 00:50:10.998176 25324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9878436561e94139ab55805b54912f38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:50:11.001672 25536 tablet_service.cc:1467] Processing CreateTablet for tablet 9878436561e94139ab55805b54912f38 (DEFAULT_TABLE table=test-table [id=2d6de86fa4534b90903ff241621aab6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:50:11.003157 25536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9878436561e94139ab55805b54912f38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:50:11.009087 25232 heartbeater.cc:503] Master 127.28.6.62:37253 was elected leader, sending a full tablet report...
I20260810 00:50:11.010555 25661 tablet_bootstrap.cc:492] T 9878436561e94139ab55805b54912f38 P e0b7d3c6bf5c401db17583908c4f5fe6: Bootstrap starting.
I20260810 00:50:11.014248 25661 tablet_bootstrap.cc:654] T 9878436561e94139ab55805b54912f38 P e0b7d3c6bf5c401db17583908c4f5fe6: Neither blocks nor log segments found. Creating new log.
I20260810 00:50:11.015548 25661 log.cc:824] T 9878436561e94139ab55805b54912f38 P e0b7d3c6bf5c401db17583908c4f5fe6: Log is configured to *not* fsync() on all Append() calls
I20260810 00:50:11.017028 25662 tablet_bootstrap.cc:492] T 9878436561e94139ab55805b54912f38 P 7cd64ac068c5454fb4c64664c89b6489: Bootstrap starting.
I20260810 00:50:11.018800 25661 tablet_bootstrap.cc:492] T 9878436561e94139ab55805b54912f38 P e0b7d3c6bf5c401db17583908c4f5fe6: No bootstrap required, opened a new log
I20260810 00:50:11.019188 25661 ts_tablet_manager.cc:1397] T 9878436561e94139ab55805b54912f38 P e0b7d3c6bf5c401db17583908c4f5fe6: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260810 00:50:11.020334 25664 tablet_bootstrap.cc:492] T 9878436561e94139ab55805b54912f38 P 195f200d5106466d98fb676fb6792d80: Bootstrap starting.
I20260810 00:50:11.020912 25662 tablet_bootstrap.cc:654] T 9878436561e94139ab55805b54912f38 P 7cd64ac068c5454fb4c64664c89b6489: Neither blocks nor log segments found. Creating new log.
I20260810 00:50:11.021955 25662 log.cc:824] T 9878436561e94139ab55805b54912f38 P 7cd64ac068c5454fb4c64664c89b6489: Log is configured to *not* fsync() on all Append() calls
I20260810 00:50:11.023912 25664 tablet_bootstrap.cc:654] T 9878436561e94139ab55805b54912f38 P 195f200d5106466d98fb676fb6792d80: Neither blocks nor log segments found. Creating new log.
I20260810 00:50:11.024600 25662 tablet_bootstrap.cc:492] T 9878436561e94139ab55805b54912f38 P 7cd64ac068c5454fb4c64664c89b6489: No bootstrap required, opened a new log
I20260810 00:50:11.024878 25662 ts_tablet_manager.cc:1397] T 9878436561e94139ab55805b54912f38 P 7cd64ac068c5454fb4c64664c89b6489: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 00:50:11.025134 25664 log.cc:824] T 9878436561e94139ab55805b54912f38 P 195f200d5106466d98fb676fb6792d80: Log is configured to *not* fsync() on all Append() calls
I20260810 00:50:11.029424 25664 tablet_bootstrap.cc:492] T 9878436561e94139ab55805b54912f38 P 195f200d5106466d98fb676fb6792d80: No bootstrap required, opened a new log
I20260810 00:50:11.029731 25664 ts_tablet_manager.cc:1397] T 9878436561e94139ab55805b54912f38 P 195f200d5106466d98fb676fb6792d80: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260810 00:50:11.029670 25661 raft_consensus.cc:348] T 9878436561e94139ab55805b54912f38 P e0b7d3c6bf5c401db17583908c4f5fe6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b7d3c6bf5c401db17583908c4f5fe6" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 40663 } } peers { permanent_uuid: "7cd64ac068c5454fb4c64664c89b6489" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 36555 } } peers { permanent_uuid: "195f200d5106466d98fb676fb6792d80" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39733 } }
I20260810 00:50:11.030164 25661 raft_consensus.cc:374] T 9878436561e94139ab55805b54912f38 P e0b7d3c6bf5c401db17583908c4f5fe6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:50:11.030282 25661 raft_consensus.cc:729] T 9878436561e94139ab55805b54912f38 P e0b7d3c6bf5c401db17583908c4f5fe6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b7d3c6bf5c401db17583908c4f5fe6, State: Initialized, Role: FOLLOWER
I20260810 00:50:11.030953 25661 consensus_queue.cc:260] T 9878436561e94139ab55805b54912f38 P e0b7d3c6bf5c401db17583908c4f5fe6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b7d3c6bf5c401db17583908c4f5fe6" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 40663 } } peers { permanent_uuid: "7cd64ac068c5454fb4c64664c89b6489" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 36555 } } peers { permanent_uuid: "195f200d5106466d98fb676fb6792d80" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39733 } }
I20260810 00:50:11.033758 25661 ts_tablet_manager.cc:1428] T 9878436561e94139ab55805b54912f38 P e0b7d3c6bf5c401db17583908c4f5fe6: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260810 00:50:11.033633 25662 raft_consensus.cc:348] T 9878436561e94139ab55805b54912f38 P 7cd64ac068c5454fb4c64664c89b6489 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b7d3c6bf5c401db17583908c4f5fe6" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 40663 } } peers { permanent_uuid: "7cd64ac068c5454fb4c64664c89b6489" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 36555 } } peers { permanent_uuid: "195f200d5106466d98fb676fb6792d80" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39733 } }
I20260810 00:50:11.034034 25662 raft_consensus.cc:374] T 9878436561e94139ab55805b54912f38 P 7cd64ac068c5454fb4c64664c89b6489 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:50:11.034139 25662 raft_consensus.cc:729] T 9878436561e94139ab55805b54912f38 P 7cd64ac068c5454fb4c64664c89b6489 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cd64ac068c5454fb4c64664c89b6489, State: Initialized, Role: FOLLOWER
I20260810 00:50:11.034632 25662 consensus_queue.cc:260] T 9878436561e94139ab55805b54912f38 P 7cd64ac068c5454fb4c64664c89b6489 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b7d3c6bf5c401db17583908c4f5fe6" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 40663 } } peers { permanent_uuid: "7cd64ac068c5454fb4c64664c89b6489" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 36555 } } peers { permanent_uuid: "195f200d5106466d98fb676fb6792d80" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39733 } }
I20260810 00:50:11.035843 25438 heartbeater.cc:503] Master 127.28.6.62:37253 was elected leader, sending a full tablet report...
I20260810 00:50:11.036545 25662 ts_tablet_manager.cc:1428] T 9878436561e94139ab55805b54912f38 P 7cd64ac068c5454fb4c64664c89b6489: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260810 00:50:11.040987 25664 raft_consensus.cc:348] T 9878436561e94139ab55805b54912f38 P 195f200d5106466d98fb676fb6792d80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b7d3c6bf5c401db17583908c4f5fe6" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 40663 } } peers { permanent_uuid: "7cd64ac068c5454fb4c64664c89b6489" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 36555 } } peers { permanent_uuid: "195f200d5106466d98fb676fb6792d80" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39733 } }
I20260810 00:50:11.041342 25664 raft_consensus.cc:374] T 9878436561e94139ab55805b54912f38 P 195f200d5106466d98fb676fb6792d80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:50:11.041455 25664 raft_consensus.cc:729] T 9878436561e94139ab55805b54912f38 P 195f200d5106466d98fb676fb6792d80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195f200d5106466d98fb676fb6792d80, State: Initialized, Role: FOLLOWER
I20260810 00:50:11.041965 25664 consensus_queue.cc:260] T 9878436561e94139ab55805b54912f38 P 195f200d5106466d98fb676fb6792d80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b7d3c6bf5c401db17583908c4f5fe6" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 40663 } } peers { permanent_uuid: "7cd64ac068c5454fb4c64664c89b6489" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 36555 } } peers { permanent_uuid: "195f200d5106466d98fb676fb6792d80" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39733 } }
I20260810 00:50:11.042927 25638 heartbeater.cc:503] Master 127.28.6.62:37253 was elected leader, sending a full tablet report...
I20260810 00:50:11.043541 25664 ts_tablet_manager.cc:1428] T 9878436561e94139ab55805b54912f38 P 195f200d5106466d98fb676fb6792d80: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260810 00:50:11.147905 25667 raft_consensus.cc:482] T 9878436561e94139ab55805b54912f38 P e0b7d3c6bf5c401db17583908c4f5fe6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:50:11.148226 25667 raft_consensus.cc:504] T 9878436561e94139ab55805b54912f38 P e0b7d3c6bf5c401db17583908c4f5fe6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b7d3c6bf5c401db17583908c4f5fe6" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 40663 } } peers { permanent_uuid: "7cd64ac068c5454fb4c64664c89b6489" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 36555 } } peers { permanent_uuid: "195f200d5106466d98fb676fb6792d80" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39733 } }
I20260810 00:50:11.149983 25667 leader_election.cc:288] T 9878436561e94139ab55805b54912f38 P e0b7d3c6bf5c401db17583908c4f5fe6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7cd64ac068c5454fb4c64664c89b6489 (127.28.6.2:36555), 195f200d5106466d98fb676fb6792d80 (127.28.6.3:39733)
I20260810 00:50:11.158780 25658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:11.152095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:34411 (local address 127.28.6.3:39733)
0810 00:50:11.152361 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:11.152370 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:11.152954 (+ 584us) server_negotiation.cc:407] Connection header received
0810 00:50:11.153161 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:11.153165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:11.153222 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:11.153316 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:50:11.153838 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:11.154530 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:11.156844 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:11.157692 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:11.157846 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:11.157948 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:11.158386 (+ 438us) server_negotiation.cc:299] Negotiation successful
0810 00:50:11.158519 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260810 00:50:11.159003 25671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:11.151634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:39733 (local address 127.28.6.1:34411)
0810 00:50:11.152841 (+ 1207us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:11.152855 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:11.153010 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:11.153346 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:11.153350 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:11.153372 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:50:11.153671 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:11.153679 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:11.154668 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:11.154674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:11.156613 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:11.156626 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:11.157902 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:11.157911 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:11.158128 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:11.158180 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 00:50:11.158335 (+ 155us) client_negotiation.cc:240] Negotiation successful
0810 00:50:11.158511 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1099,"mutex_wait_us":107,"thread_start_us":914,"threads_started":1}
I20260810 00:50:11.160080 25670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:11.150302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:36555 (local address 127.28.6.1:60827)
0810 00:50:11.151655 (+ 1353us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:11.151706 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:11.151967 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:11.152631 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:11.152638 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:11.152688 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:50:11.153185 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:11.153202 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:11.154927 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:11.154936 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:11.157601 (+ 2665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:11.157621 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:11.159354 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:11.159360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:11.159539 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:11.159577 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 00:50:11.159679 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 00:50:11.159796 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1136,"thread_start_us":760,"threads_started":1}
I20260810 00:50:11.160095 25657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:11.150482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:60827 (local address 127.28.6.2:36555)
0810 00:50:11.150710 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:11.150720 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:11.151823 (+ 1103us) server_negotiation.cc:407] Connection header received
0810 00:50:11.152173 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:11.152181 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:11.152296 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:11.152381 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:50:11.153414 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:11.154689 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:11.157819 (+ 3130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:11.158991 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:11.159102 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:11.159186 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:11.159707 (+ 521us) server_negotiation.cc:299] Negotiation successful
0810 00:50:11.159858 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 00:50:11.160411 25565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9878436561e94139ab55805b54912f38" candidate_uuid: "e0b7d3c6bf5c401db17583908c4f5fe6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195f200d5106466d98fb676fb6792d80" is_pre_election: true
I20260810 00:50:11.161113 25565 raft_consensus.cc:2454] T 9878436561e94139ab55805b54912f38 P 195f200d5106466d98fb676fb6792d80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b7d3c6bf5c401db17583908c4f5fe6 in term 0.
I20260810 00:50:11.161206 25356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9878436561e94139ab55805b54912f38" candidate_uuid: "e0b7d3c6bf5c401db17583908c4f5fe6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cd64ac068c5454fb4c64664c89b6489" is_pre_election: true
I20260810 00:50:11.161803 25356 raft_consensus.cc:2454] T 9878436561e94139ab55805b54912f38 P 7cd64ac068c5454fb4c64664c89b6489 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0b7d3c6bf5c401db17583908c4f5fe6 in term 0.
I20260810 00:50:11.161919 25055 leader_election.cc:302] T 9878436561e94139ab55805b54912f38 P e0b7d3c6bf5c401db17583908c4f5fe6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195f200d5106466d98fb676fb6792d80, e0b7d3c6bf5c401db17583908c4f5fe6; no voters:
I20260810 00:50:11.162335 25667 raft_consensus.cc:2781] T 9878436561e94139ab55805b54912f38 P e0b7d3c6bf5c401db17583908c4f5fe6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 00:50:11.162485 25667 raft_consensus.cc:482] T 9878436561e94139ab55805b54912f38 P e0b7d3c6bf5c401db17583908c4f5fe6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 00:50:11.162606 25667 raft_consensus.cc:3037] T 9878436561e94139ab55805b54912f38 P e0b7d3c6bf5c401db17583908c4f5fe6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:50:11.164821 25667 raft_consensus.cc:504] T 9878436561e94139ab55805b54912f38 P e0b7d3c6bf5c401db17583908c4f5fe6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b7d3c6bf5c401db17583908c4f5fe6" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 40663 } } peers { permanent_uuid: "7cd64ac068c5454fb4c64664c89b6489" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 36555 } } peers { permanent_uuid: "195f200d5106466d98fb676fb6792d80" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39733 } }
I20260810 00:50:11.165506 25667 leader_election.cc:288] T 9878436561e94139ab55805b54912f38 P e0b7d3c6bf5c401db17583908c4f5fe6 [CANDIDATE]: Term 1 election: Requested vote from peers 7cd64ac068c5454fb4c64664c89b6489 (127.28.6.2:36555), 195f200d5106466d98fb676fb6792d80 (127.28.6.3:39733)
I20260810 00:50:11.165978 25565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9878436561e94139ab55805b54912f38" candidate_uuid: "e0b7d3c6bf5c401db17583908c4f5fe6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195f200d5106466d98fb676fb6792d80"
I20260810 00:50:11.165966 25356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9878436561e94139ab55805b54912f38" candidate_uuid: "e0b7d3c6bf5c401db17583908c4f5fe6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cd64ac068c5454fb4c64664c89b6489"
I20260810 00:50:11.166172 25356 raft_consensus.cc:3037] T 9878436561e94139ab55805b54912f38 P 7cd64ac068c5454fb4c64664c89b6489 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:50:11.166172 25565 raft_consensus.cc:3037] T 9878436561e94139ab55805b54912f38 P 195f200d5106466d98fb676fb6792d80 [term 0 FOLLOWER]: Advancing to term 1
W20260810 00:50:11.167896 25639 tablet.cc:2369] T 9878436561e94139ab55805b54912f38 P 195f200d5106466d98fb676fb6792d80: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:50:11.169545 25565 raft_consensus.cc:2454] T 9878436561e94139ab55805b54912f38 P 195f200d5106466d98fb676fb6792d80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b7d3c6bf5c401db17583908c4f5fe6 in term 1.
I20260810 00:50:11.169853 25356 raft_consensus.cc:2454] T 9878436561e94139ab55805b54912f38 P 7cd64ac068c5454fb4c64664c89b6489 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0b7d3c6bf5c401db17583908c4f5fe6 in term 1.
I20260810 00:50:11.169937 25055 leader_election.cc:302] T 9878436561e94139ab55805b54912f38 P e0b7d3c6bf5c401db17583908c4f5fe6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195f200d5106466d98fb676fb6792d80, e0b7d3c6bf5c401db17583908c4f5fe6; no voters:
I20260810 00:50:11.170197 25667 raft_consensus.cc:2781] T 9878436561e94139ab55805b54912f38 P e0b7d3c6bf5c401db17583908c4f5fe6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:50:11.170712 25667 raft_consensus.cc:686] T 9878436561e94139ab55805b54912f38 P e0b7d3c6bf5c401db17583908c4f5fe6 [term 1 LEADER]: Becoming Leader. State: Replica: e0b7d3c6bf5c401db17583908c4f5fe6, State: Running, Role: LEADER
I20260810 00:50:11.171448 25667 consensus_queue.cc:237] T 9878436561e94139ab55805b54912f38 P e0b7d3c6bf5c401db17583908c4f5fe6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b7d3c6bf5c401db17583908c4f5fe6" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 40663 } } peers { permanent_uuid: "7cd64ac068c5454fb4c64664c89b6489" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 36555 } } peers { permanent_uuid: "195f200d5106466d98fb676fb6792d80" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39733 } }
I20260810 00:50:11.176687 24944 catalog_manager.cc:5495] T 9878436561e94139ab55805b54912f38 P e0b7d3c6bf5c401db17583908c4f5fe6 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b7d3c6bf5c401db17583908c4f5fe6 (127.28.6.1). New cstate: current_term: 1 leader_uuid: "e0b7d3c6bf5c401db17583908c4f5fe6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0b7d3c6bf5c401db17583908c4f5fe6" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 40663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7cd64ac068c5454fb4c64664c89b6489" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 36555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "195f200d5106466d98fb676fb6792d80" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39733 } health_report { overall_health: UNKNOWN } } }
W20260810 00:50:11.204581 25439 tablet.cc:2369] T 9878436561e94139ab55805b54912f38 P 7cd64ac068c5454fb4c64664c89b6489: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 00:50:11.237404 25233 tablet.cc:2369] T 9878436561e94139ab55805b54912f38 P e0b7d3c6bf5c401db17583908c4f5fe6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:50:11.238832 25655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:11.233072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52490 (local address 127.28.6.1:40663)
0810 00:50:11.233323 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:11.233329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:11.233349 (+ 20us) server_negotiation.cc:407] Connection header received
0810 00:50:11.233544 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:11.233548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:11.233647 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:11.233803 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:50:11.234182 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:11.234891 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:11.236310 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:11.236736 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:11.236870 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:11.237530 (+ 660us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:50:11.238124 (+ 594us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:50:11.238350 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:11.238526 (+ 176us) server_negotiation.cc:299] Negotiation successful
0810 00:50:11.238570 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260810 00:50:11.248799 25356 raft_consensus.cc:1261] T 9878436561e94139ab55805b54912f38 P 7cd64ac068c5454fb4c64664c89b6489 [term 1 FOLLOWER]: Refusing update from remote peer e0b7d3c6bf5c401db17583908c4f5fe6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:50:11.248799 25565 raft_consensus.cc:1261] T 9878436561e94139ab55805b54912f38 P 195f200d5106466d98fb676fb6792d80 [term 1 FOLLOWER]: Refusing update from remote peer e0b7d3c6bf5c401db17583908c4f5fe6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:50:11.249806 25672 consensus_queue.cc:1035] T 9878436561e94139ab55805b54912f38 P e0b7d3c6bf5c401db17583908c4f5fe6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cd64ac068c5454fb4c64664c89b6489" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 36555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:50:11.250298 25667 consensus_queue.cc:1035] T 9878436561e94139ab55805b54912f38 P e0b7d3c6bf5c401db17583908c4f5fe6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195f200d5106466d98fb676fb6792d80" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:50:11.264290 25676 mvcc.cc:205] Tried to move back new op lower bound from 7316779054064300032 to 7316779053762211840. Current Snapshot: MvccSnapshot[applied={T|T < 7316779054064300032}]
I20260810 00:50:11.281530 24944 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:50:11.281919 24944 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:50:11.286331 24944 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ba68d1b46992431ebf99b05cf2daede6: Sending DeleteTablet for 3 replicas of tablet 9878436561e94139ab55805b54912f38
I20260810 00:50:11.287494 25119 tablet_service.cc:1514] Processing DeleteTablet for tablet 9878436561e94139ab55805b54912f38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:50:11 UTC) from {username='slave'} at 127.0.0.1:52478
I20260810 00:50:11.290678 25324 tablet_service.cc:1514] Processing DeleteTablet for tablet 9878436561e94139ab55805b54912f38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:50:11 UTC) from {username='slave'} at 127.0.0.1:46184
I20260810 00:50:11.292340 25322 tablet_service.cc:1514] Processing DeleteTablet for tablet 9878436561e94139ab55805b54912f38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:50:11 UTC) from {username='slave'} at 127.0.0.1:46184
I20260810 00:50:11.293717 25536 tablet_service.cc:1514] Processing DeleteTablet for tablet 9878436561e94139ab55805b54912f38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:50:11 UTC) from {username='slave'} at 127.0.0.1:56260
I20260810 00:50:11.302911 25686 tablet_replica.cc:330] T 9878436561e94139ab55805b54912f38 P e0b7d3c6bf5c401db17583908c4f5fe6: stopping tablet replica
I20260810 00:50:11.303045 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 25030
I20260810 00:50:11.304006 25686 raft_consensus.cc:2229] T 9878436561e94139ab55805b54912f38 P e0b7d3c6bf5c401db17583908c4f5fe6 [term 1 LEADER]: Raft consensus shutting down.
I20260810 00:50:11.304683 25686 raft_consensus.cc:2258] T 9878436561e94139ab55805b54912f38 P e0b7d3c6bf5c401db17583908c4f5fe6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:50:11.308408 25686 ts_tablet_manager.cc:1905] T 9878436561e94139ab55805b54912f38 P e0b7d3c6bf5c401db17583908c4f5fe6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:50:11.324566 25692 tablet_replica.cc:330] T 9878436561e94139ab55805b54912f38 P 195f200d5106466d98fb676fb6792d80: stopping tablet replica
I20260810 00:50:11.326853 25692 raft_consensus.cc:2229] T 9878436561e94139ab55805b54912f38 P 195f200d5106466d98fb676fb6792d80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:50:11.329285 25686 ts_tablet_manager.cc:1918] T 9878436561e94139ab55805b54912f38 P e0b7d3c6bf5c401db17583908c4f5fe6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:50:11.329538 25686 log.cc:1196] T 9878436561e94139ab55805b54912f38 P e0b7d3c6bf5c401db17583908c4f5fe6: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-0/wal/wals/9878436561e94139ab55805b54912f38
I20260810 00:50:11.330163 25686 ts_tablet_manager.cc:1939] T 9878436561e94139ab55805b54912f38 P e0b7d3c6bf5c401db17583908c4f5fe6: Deleting consensus metadata
I20260810 00:50:11.330858 25690 tablet_replica.cc:330] T 9878436561e94139ab55805b54912f38 P 7cd64ac068c5454fb4c64664c89b6489: stopping tablet replica
I20260810 00:50:11.331028 25692 raft_consensus.cc:2258] T 9878436561e94139ab55805b54912f38 P 195f200d5106466d98fb676fb6792d80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:50:11.332139 24922 catalog_manager.cc:4841] TS e0b7d3c6bf5c401db17583908c4f5fe6 (127.28.6.1:40663): tablet 9878436561e94139ab55805b54912f38 (table test-table [id=2d6de86fa4534b90903ff241621aab6c]) successfully deleted
I20260810 00:50:11.331735 25690 raft_consensus.cc:2229] T 9878436561e94139ab55805b54912f38 P 7cd64ac068c5454fb4c64664c89b6489 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:50:11.334443 25690 raft_consensus.cc:2258] T 9878436561e94139ab55805b54912f38 P 7cd64ac068c5454fb4c64664c89b6489 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:50:11.334619 25692 ts_tablet_manager.cc:1905] T 9878436561e94139ab55805b54912f38 P 195f200d5106466d98fb676fb6792d80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:50:11.336699 25655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:11.304550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52506 (local address 127.28.6.1:40663)
0810 00:50:11.304728 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:11.304733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:11.325276 (+ 20543us) server_negotiation.cc:407] Connection header received
0810 00:50:11.325931 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:11.325941 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:11.326075 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:11.326167 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:11.327779 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:11.329360 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:11.332381 (+ 3021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:11.333037 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:11.333148 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:11.335012 (+ 1864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:11.335025 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:11.335046 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:11.335828 (+ 782us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:11.335879 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:11.335906 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:11.335992 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:11.336221 (+ 229us) server_negotiation.cc:299] Negotiation successful
0810 00:50:11.336328 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 00:50:11.337944 25224 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:50:11.342770 25692 ts_tablet_manager.cc:1918] T 9878436561e94139ab55805b54912f38 P 195f200d5106466d98fb676fb6792d80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:50:11.343027 25692 log.cc:1196] T 9878436561e94139ab55805b54912f38 P 195f200d5106466d98fb676fb6792d80: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-2/wal/wals/9878436561e94139ab55805b54912f38
I20260810 00:50:11.345037 25692 ts_tablet_manager.cc:1939] T 9878436561e94139ab55805b54912f38 P 195f200d5106466d98fb676fb6792d80: Deleting consensus metadata
I20260810 00:50:11.347076 24919 catalog_manager.cc:4841] TS 195f200d5106466d98fb676fb6792d80 (127.28.6.3:39733): tablet 9878436561e94139ab55805b54912f38 (table test-table [id=2d6de86fa4534b90903ff241621aab6c]) successfully deleted
I20260810 00:50:11.347674 25690 ts_tablet_manager.cc:1905] T 9878436561e94139ab55805b54912f38 P 7cd64ac068c5454fb4c64664c89b6489: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:50:11.352293 25690 ts_tablet_manager.cc:1918] T 9878436561e94139ab55805b54912f38 P 7cd64ac068c5454fb4c64664c89b6489: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:50:11.352473 25690 log.cc:1196] T 9878436561e94139ab55805b54912f38 P 7cd64ac068c5454fb4c64664c89b6489: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786322687226232-28696-0/minicluster-data/ts-1/wal/wals/9878436561e94139ab55805b54912f38
I20260810 00:50:11.353029 25690 ts_tablet_manager.cc:1939] T 9878436561e94139ab55805b54912f38 P 7cd64ac068c5454fb4c64664c89b6489: Deleting consensus metadata
I20260810 00:50:11.354483 24919 catalog_manager.cc:4841] TS 7cd64ac068c5454fb4c64664c89b6489 (127.28.6.2:36555): tablet 9878436561e94139ab55805b54912f38 (table test-table [id=2d6de86fa4534b90903ff241621aab6c]) successfully deleted
W20260810 00:50:11.355062 24919 catalog_manager.cc:4808] TS 7cd64ac068c5454fb4c64664c89b6489 (127.28.6.2:36555): delete failed for tablet 9878436561e94139ab55805b54912f38 because the tablet was not found. No further retry: Not found: Tablet not found: 9878436561e94139ab55805b54912f38
I20260810 00:50:11.458671 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 25030
I20260810 00:50:11.492355 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 25237
I20260810 00:50:11.499666 25657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:11.493165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46194 (local address 127.28.6.2:36555)
0810 00:50:11.493402 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:11.493408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:11.493429 (+ 21us) server_negotiation.cc:407] Connection header received
0810 00:50:11.493582 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:11.493587 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:11.493676 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:11.493788 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:11.494266 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:11.495135 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:11.496412 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:11.496829 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:11.496969 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:11.498142 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:11.498151 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:11.498165 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:11.498813 (+ 648us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:11.498863 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:11.498883 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:11.499105 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:11.499240 (+ 135us) server_negotiation.cc:299] Negotiation successful
0810 00:50:11.499358 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 00:50:11.500560 25426 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:50:11.590569 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 25237
I20260810 00:50:11.616477 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 25441
I20260810 00:50:11.623751 25658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:11.617281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56262 (local address 127.28.6.3:39733)
0810 00:50:11.617484 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:11.617491 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:11.617511 (+ 20us) server_negotiation.cc:407] Connection header received
0810 00:50:11.617717 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:11.617723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:11.617825 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:11.617936 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:11.618385 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:11.619194 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:11.620542 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:11.620961 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:11.621104 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:11.622235 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:11.622246 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:11.622262 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:11.622846 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:11.622967 (+ 121us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:11.622998 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:11.623111 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:11.623296 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 00:50:11.623416 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 00:50:11.624404 25632 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:50:11.711951 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 25441
I20260810 00:50:11.741434 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 24897
I20260810 00:50:11.742532 24987 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:50:11.849872 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 24897
2026-08-10T00:50:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2971 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:50:11 dist-test-slave-1nl6 krb5kdc[25701](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:50:11 dist-test-slave-1nl6 krb5kdc[25701](info): set up 2 sockets
Aug 10 00:50:11 dist-test-slave-1nl6 krb5kdc[25701](info): commencing operation
krb5kdc: starting...
W20260810 00:50:13.177146 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.275s user 0.001s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:50:13 dist-test-slave-1nl6 krb5kdc[25701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:50:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:50:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.6.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
I20260810 00:50:13.454125 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:39939
--webserver_interface=127.28.6.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:40671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:39939
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:50:13.783604 25726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:50:13.783927 25726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:50:13.784106 25726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:50:13.790974 25726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:50:13.791110 25726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:50:13.791217 25726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:50:13.791325 25726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:50:13.791412 25726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:50:13.800475 25726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:40671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:39939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:39939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:50:13.801699 25726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:50:13.802986 25726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:50:13.814554 25736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:13.814594 25738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:13.814615 25744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:50:13.815646 25726 server_base.cc:1034] running on GCE node
I20260810 00:50:13.816402 25726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:50:13.817698 25726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:50:13.818981 25726 hybrid_clock.cc:648] HybridClock initialized: now 1786323013818911 us; error 59 us; skew 500 ppm
I20260810 00:50:13.819343 25726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:50:13 dist-test-slave-1nl6 krb5kdc[25701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323013, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:50:13.822425 25726 init.cc:377] Logged in from keytab as kudu/127.28.6.62@KRBTEST.COM (short username kudu)
I20260810 00:50:13.823823 25726 webserver.cc:460] Webserver started at http://127.28.6.62:41055/ using document root <none> and password file <none>
I20260810 00:50:13.824468 25726 fs_manager.cc:359] Metadata directory not provided
I20260810 00:50:13.824615 25726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:50:13.824941 25726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:50:13.827925 25726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "729b5a3ca8b04c4aa956a4099c02253b"
format_stamp: "Formatted at 2026-08-10 00:50:13 on dist-test-slave-1nl6"
server_key: "ea8075a04ab3736f2759c8497c09446f"
server_key_iv: "3e88eb5879b29ce326efd422da1d8015"
server_key_version: "encryptionkey@0"
I20260810 00:50:13.828706 25726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "729b5a3ca8b04c4aa956a4099c02253b"
format_stamp: "Formatted at 2026-08-10 00:50:13 on dist-test-slave-1nl6"
server_key: "ea8075a04ab3736f2759c8497c09446f"
server_key_iv: "3e88eb5879b29ce326efd422da1d8015"
server_key_version: "encryptionkey@0"
I20260810 00:50:13.833649 25726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 00:50:13.836953 25755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:13.838193 25726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 00:50:13.838400 25726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "729b5a3ca8b04c4aa956a4099c02253b"
format_stamp: "Formatted at 2026-08-10 00:50:13 on dist-test-slave-1nl6"
server_key: "ea8075a04ab3736f2759c8497c09446f"
server_key_iv: "3e88eb5879b29ce326efd422da1d8015"
server_key_version: "encryptionkey@0"
I20260810 00:50:13.838600 25726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:50:13.855237 25726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:50:13.859566 25726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:50:13.859859 25726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:50:13.878628 25726 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:39939
I20260810 00:50:13.878683 25823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:39939 every 8 connection(s)
I20260810 00:50:13.880167 25726 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:50:13.883878 25824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:50:13.889501 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 25726
I20260810 00:50:13.889703 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:50:13.890275 28696 external_mini_cluster.cc:1428] Setting key c0aa5f8a609959450d73e26356236e45
I20260810 00:50:13.895577 25824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 729b5a3ca8b04c4aa956a4099c02253b: Bootstrap starting.
I20260810 00:50:13.900063 25824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 729b5a3ca8b04c4aa956a4099c02253b: Neither blocks nor log segments found. Creating new log.
I20260810 00:50:13.901475 25824 log.cc:824] T 00000000000000000000000000000000 P 729b5a3ca8b04c4aa956a4099c02253b: Log is configured to *not* fsync() on all Append() calls
I20260810 00:50:13.905174 25824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 729b5a3ca8b04c4aa956a4099c02253b: No bootstrap required, opened a new log
Aug 10 00:50:13 dist-test-slave-1nl6 krb5kdc[25701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:50:13.915926 25824 raft_consensus.cc:348] T 00000000000000000000000000000000 P 729b5a3ca8b04c4aa956a4099c02253b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729b5a3ca8b04c4aa956a4099c02253b" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39939 } }
I20260810 00:50:13.916385 25824 raft_consensus.cc:374] T 00000000000000000000000000000000 P 729b5a3ca8b04c4aa956a4099c02253b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:50:13.916497 25824 raft_consensus.cc:729] T 00000000000000000000000000000000 P 729b5a3ca8b04c4aa956a4099c02253b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 729b5a3ca8b04c4aa956a4099c02253b, State: Initialized, Role: FOLLOWER
I20260810 00:50:13.916757 25832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:13.892397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34808 (local address 127.28.6.62:39939)
0810 00:50:13.893304 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:13.893315 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:13.893367 (+ 52us) server_negotiation.cc:407] Connection header received
0810 00:50:13.895183 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:13.895211 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:13.895608 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:13.895970 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:13.898156 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:13.899553 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:13.901650 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:13.902755 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:13.902991 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:13.907944 (+ 4953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:13.907954 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:13.907974 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:50:13.908029 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:13.912099 (+ 4070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:13.912844 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:13.912852 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:13.912860 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:13.913060 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:13.913491 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:13.913501 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:13.913504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:13.914242 (+ 738us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:13.914541 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:13.914963 (+ 422us) server_negotiation.cc:299] Negotiation successful
0810 00:50:13.915387 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":445,"threads_started":1}
I20260810 00:50:13.917176 25824 consensus_queue.cc:260] T 00000000000000000000000000000000 P 729b5a3ca8b04c4aa956a4099c02253b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729b5a3ca8b04c4aa956a4099c02253b" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39939 } }
I20260810 00:50:13.917419 25824 raft_consensus.cc:388] T 00000000000000000000000000000000 P 729b5a3ca8b04c4aa956a4099c02253b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:50:13.917569 25824 raft_consensus.cc:482] T 00000000000000000000000000000000 P 729b5a3ca8b04c4aa956a4099c02253b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:50:13.917760 25824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 729b5a3ca8b04c4aa956a4099c02253b [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:50:13.920163 25824 raft_consensus.cc:504] T 00000000000000000000000000000000 P 729b5a3ca8b04c4aa956a4099c02253b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729b5a3ca8b04c4aa956a4099c02253b" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39939 } }
I20260810 00:50:13.920643 25824 leader_election.cc:302] T 00000000000000000000000000000000 P 729b5a3ca8b04c4aa956a4099c02253b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 729b5a3ca8b04c4aa956a4099c02253b; no voters:
I20260810 00:50:13.921214 25824 leader_election.cc:288] T 00000000000000000000000000000000 P 729b5a3ca8b04c4aa956a4099c02253b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:50:13.921346 25834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 729b5a3ca8b04c4aa956a4099c02253b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:50:13.922652 25834 raft_consensus.cc:686] T 00000000000000000000000000000000 P 729b5a3ca8b04c4aa956a4099c02253b [term 1 LEADER]: Becoming Leader. State: Replica: 729b5a3ca8b04c4aa956a4099c02253b, State: Running, Role: LEADER
I20260810 00:50:13.923272 25834 consensus_queue.cc:237] T 00000000000000000000000000000000 P 729b5a3ca8b04c4aa956a4099c02253b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729b5a3ca8b04c4aa956a4099c02253b" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39939 } }
I20260810 00:50:13.923584 25824 sys_catalog.cc:564] T 00000000000000000000000000000000 P 729b5a3ca8b04c4aa956a4099c02253b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:50:13.926173 25836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 729b5a3ca8b04c4aa956a4099c02253b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "729b5a3ca8b04c4aa956a4099c02253b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729b5a3ca8b04c4aa956a4099c02253b" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39939 } } }
I20260810 00:50:13.926182 25839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 729b5a3ca8b04c4aa956a4099c02253b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 729b5a3ca8b04c4aa956a4099c02253b. Latest consensus state: current_term: 1 leader_uuid: "729b5a3ca8b04c4aa956a4099c02253b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729b5a3ca8b04c4aa956a4099c02253b" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 39939 } } }
I20260810 00:50:13.926615 25839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 729b5a3ca8b04c4aa956a4099c02253b [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:13.926615 25836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 729b5a3ca8b04c4aa956a4099c02253b [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:13.931890 25844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:50:13.937427 25844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:50:13.947268 25844 catalog_manager.cc:1347] Generated new cluster ID: e7e23d01d4324b8e8ad96657dfaf7049
I20260810 00:50:13.947386 25844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:50:13.961387 25844 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:50:13.962683 25844 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:50:13.973029 25844 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 729b5a3ca8b04c4aa956a4099c02253b: Generated new TSK 0
I20260810 00:50:13.973805 25844 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.6.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab.
Entry for principal kudu/127.28.6.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab.
I20260810 00:50:14.053992 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.1:0
--local_ip_for_outbound_sockets=127.28.6.1
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:39939
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:40671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:50:14.392613 25865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:50:14.392921 25865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:50:14.393098 25865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:50:14.400005 25865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:50:14.400146 25865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:50:14.400336 25865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.1
I20260810 00:50:14.409403 25865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:40671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.1.keytab
--principal=kudu/127.28.6.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:39939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.1
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:50:14.410625 25865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:50:14.411936 25865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:50:14.424824 25876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:14.424772 25879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:14.425287 25875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:50:14.425865 25865 server_base.cc:1034] running on GCE node
I20260810 00:50:14.426445 25865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:50:14.427397 25865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:50:14.428628 25865 hybrid_clock.cc:648] HybridClock initialized: now 1786323014428573 us; error 49 us; skew 500 ppm
I20260810 00:50:14.428938 25865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:50:14 dist-test-slave-1nl6 krb5kdc[25701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323014, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:50:14.432293 25865 init.cc:377] Logged in from keytab as kudu/127.28.6.1@KRBTEST.COM (short username kudu)
I20260810 00:50:14.433696 25865 webserver.cc:460] Webserver started at http://127.28.6.1:35747/ using document root <none> and password file <none>
I20260810 00:50:14.434356 25865 fs_manager.cc:359] Metadata directory not provided
I20260810 00:50:14.434491 25865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:50:14.434811 25865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:50:14.437613 25865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-0/data/instance:
uuid: "c457caac937b4966903cb1057679ef38"
format_stamp: "Formatted at 2026-08-10 00:50:14 on dist-test-slave-1nl6"
server_key: "864549312b94d1c5b713a98758cc5db6"
server_key_iv: "84785c9e58022a6045cb2019b5b04328"
server_key_version: "encryptionkey@0"
I20260810 00:50:14.438371 25865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance:
uuid: "c457caac937b4966903cb1057679ef38"
format_stamp: "Formatted at 2026-08-10 00:50:14 on dist-test-slave-1nl6"
server_key: "864549312b94d1c5b713a98758cc5db6"
server_key_iv: "84785c9e58022a6045cb2019b5b04328"
server_key_version: "encryptionkey@0"
I20260810 00:50:14.443550 25865 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 00:50:14.446767 25893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:14.448026 25865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 00:50:14.448206 25865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-0/wal
uuid: "c457caac937b4966903cb1057679ef38"
format_stamp: "Formatted at 2026-08-10 00:50:14 on dist-test-slave-1nl6"
server_key: "864549312b94d1c5b713a98758cc5db6"
server_key_iv: "84785c9e58022a6045cb2019b5b04328"
server_key_version: "encryptionkey@0"
I20260810 00:50:14.448411 25865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:50:14.467674 25865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:50:14.471899 25865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:50:14.472222 25865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:50:14.473172 25865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:50:14.474459 25865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:50:14.474534 25865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:14.474712 25865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:50:14.474766 25865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:14.507098 25865 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.1:33997
I20260810 00:50:14.507129 26075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.1:33997 every 8 connection(s)
I20260810 00:50:14.508363 25865 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
I20260810 00:50:14.513926 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 25865
I20260810 00:50:14.514125 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance
I20260810 00:50:14.514695 28696 external_mini_cluster.cc:1428] Setting key ac6f631b01befbef9d3983ad72e6779c
Aug 10 00:50:14 dist-test-slave-1nl6 krb5kdc[25701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323014, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.1@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:50:14.529808 26083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:14.510801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:47993 (local address 127.28.6.62:39939)
0810 00:50:14.511307 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:14.511314 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:14.512403 (+ 1089us) server_negotiation.cc:407] Connection header received
0810 00:50:14.514891 (+ 2488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:14.514919 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:14.515018 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:14.515144 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:14.516958 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:14.518045 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:14.519666 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:14.520371 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:14.520520 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:14.523906 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:14.523915 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:14.523922 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:50:14.523968 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:14.526776 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:14.527517 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:14.527529 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:14.527532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:14.527650 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:14.528095 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:14.528100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:14.528110 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:14.528362 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:14.528523 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:14.529183 (+ 660us) server_negotiation.cc:299] Negotiation successful
0810 00:50:14.529410 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":257,"threads_started":1}
I20260810 00:50:14.530704 26084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:14.511320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39939 (local address 127.28.6.1:47993)
0810 00:50:14.512198 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:14.512261 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:14.514574 (+ 2313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:14.515395 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:14.515403 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:14.516045 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:50:14.516728 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:14.516757 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:14.518223 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:14.518227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:14.519518 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:14.519531 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:14.520580 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:14.520585 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:14.520769 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:14.521459 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:50:14.521486 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:50:14.523663 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:50:14.526950 (+ 3287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:50:14.526960 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:50:14.526975 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:50:14.527385 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:50:14.527766 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:50:14.527771 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:50:14.527775 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:50:14.527972 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:50:14.528496 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:50:14.528504 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:50:14.528812 (+ 308us) client_negotiation.cc:769] Sending connection context
0810 00:50:14.529072 (+ 260us) client_negotiation.cc:240] Negotiation successful
0810 00:50:14.529573 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":312,"threads_started":1}
I20260810 00:50:14.532320 26076 heartbeater.cc:347] Connected to a master server at 127.28.6.62:39939
I20260810 00:50:14.532626 26076 heartbeater.cc:464] Registering TS with master...
I20260810 00:50:14.533532 26076 heartbeater.cc:511] Master 127.28.6.62:39939 requested a full tablet report, sending...
I20260810 00:50:14.535959 25777 ts_manager.cc:194] Registered new tserver with Master: c457caac937b4966903cb1057679ef38 (127.28.6.1:33997)
I20260810 00:50:14.538220 25777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.1@KRBTEST.COM'} at 127.28.6.1:47993
WARNING: no policy specified for kudu/127.28.6.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab.
Entry for principal kudu/127.28.6.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab.
I20260810 00:50:14.580294 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.2:0
--local_ip_for_outbound_sockets=127.28.6.2
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:39939
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:40671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:50:14.918648 26088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:50:14.919011 26088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:50:14.919175 26088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:50:14.926357 26088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:50:14.926493 26088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:50:14.926671 26088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.2
I20260810 00:50:14.936682 26088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:40671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.2.keytab
--principal=kudu/127.28.6.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:39939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.2
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:50:14.937947 26088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:50:14.939363 26088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:50:14.952494 26097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:14.952548 26100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:14.952966 26098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:50:14.953649 26088 server_base.cc:1034] running on GCE node
I20260810 00:50:14.954221 26088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:50:14.955082 26088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:50:14.956347 26088 hybrid_clock.cc:648] HybridClock initialized: now 1786323014956311 us; error 45 us; skew 500 ppm
I20260810 00:50:14.956684 26088 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:50:14 dist-test-slave-1nl6 krb5kdc[25701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323014, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:50:14.959986 26088 init.cc:377] Logged in from keytab as kudu/127.28.6.2@KRBTEST.COM (short username kudu)
I20260810 00:50:14.961494 26088 webserver.cc:460] Webserver started at http://127.28.6.2:42867/ using document root <none> and password file <none>
I20260810 00:50:14.962170 26088 fs_manager.cc:359] Metadata directory not provided
I20260810 00:50:14.962307 26088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:50:14.962639 26088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:50:14.965610 26088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-1/data/instance:
uuid: "52ddf38211c84321808ce4ae4ed23dd1"
format_stamp: "Formatted at 2026-08-10 00:50:14 on dist-test-slave-1nl6"
server_key: "2ef56a30b2e3933df07f90b29e0c8213"
server_key_iv: "c1f81386b70f5875e194deac97723161"
server_key_version: "encryptionkey@0"
I20260810 00:50:14.966408 26088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance:
uuid: "52ddf38211c84321808ce4ae4ed23dd1"
format_stamp: "Formatted at 2026-08-10 00:50:14 on dist-test-slave-1nl6"
server_key: "2ef56a30b2e3933df07f90b29e0c8213"
server_key_iv: "c1f81386b70f5875e194deac97723161"
server_key_version: "encryptionkey@0"
I20260810 00:50:14.971585 26088 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260810 00:50:14.975088 26112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:14.976315 26088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 00:50:14.976496 26088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-1/wal
uuid: "52ddf38211c84321808ce4ae4ed23dd1"
format_stamp: "Formatted at 2026-08-10 00:50:14 on dist-test-slave-1nl6"
server_key: "2ef56a30b2e3933df07f90b29e0c8213"
server_key_iv: "c1f81386b70f5875e194deac97723161"
server_key_version: "encryptionkey@0"
I20260810 00:50:14.976688 26088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:50:14.994377 26088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:50:14.998114 26088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:50:14.998399 26088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:50:14.999400 26088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:50:15.000707 26088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:50:15.000847 26088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:15.000983 26088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:50:15.001055 26088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:15.034515 26088 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.2:36115
I20260810 00:50:15.034600 26289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.2:36115 every 8 connection(s)
I20260810 00:50:15.035862 26088 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
I20260810 00:50:15.045498 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 26088
I20260810 00:50:15.045678 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance
I20260810 00:50:15.046128 28696 external_mini_cluster.cc:1428] Setting key 04df401a98c9b917da55ba98b426a839
Aug 10 00:50:15 dist-test-slave-1nl6 krb5kdc[25701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323014, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.2@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:50:15.058039 26293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:15.038422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:56443 (local address 127.28.6.62:39939)
0810 00:50:15.038859 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:15.038863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:15.039936 (+ 1073us) server_negotiation.cc:407] Connection header received
0810 00:50:15.041810 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:15.041816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:15.041922 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:15.042051 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:15.043695 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:15.044423 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:15.046077 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:15.046559 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:15.046711 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:15.050704 (+ 3993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:15.050713 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:15.050720 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:50:15.050774 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:15.054597 (+ 3823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:15.055434 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:15.055442 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:15.055448 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:15.055610 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:15.056041 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:15.056048 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:15.056054 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:15.056436 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:15.056590 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:15.057270 (+ 680us) server_negotiation.cc:299] Negotiation successful
0810 00:50:15.057530 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":216,"threads_started":1}
I20260810 00:50:15.058651 26294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:15.039037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39939 (local address 127.28.6.2:56443)
0810 00:50:15.039773 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:15.039820 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:15.041500 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:15.042296 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:15.042304 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:15.042893 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:50:15.043517 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:15.043536 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:15.044585 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:15.044589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:15.045932 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:15.045946 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:15.046803 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:15.046809 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:15.047125 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:15.047837 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:50:15.047890 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:50:15.050470 (+ 2580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:50:15.054752 (+ 4282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:50:15.054760 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:50:15.054774 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:50:15.055297 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:50:15.055722 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:50:15.055726 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:50:15.055728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:50:15.055924 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:50:15.056557 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:50:15.056562 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:50:15.056898 (+ 336us) client_negotiation.cc:769] Sending connection context
0810 00:50:15.057126 (+ 228us) client_negotiation.cc:240] Negotiation successful
0810 00:50:15.057516 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":281,"threads_started":1}
I20260810 00:50:15.060271 26290 heartbeater.cc:347] Connected to a master server at 127.28.6.62:39939
I20260810 00:50:15.060587 26290 heartbeater.cc:464] Registering TS with master...
I20260810 00:50:15.061349 26290 heartbeater.cc:511] Master 127.28.6.62:39939 requested a full tablet report, sending...
I20260810 00:50:15.062804 25777 ts_manager.cc:194] Registered new tserver with Master: 52ddf38211c84321808ce4ae4ed23dd1 (127.28.6.2:36115)
I20260810 00:50:15.063665 25777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.2@KRBTEST.COM'} at 127.28.6.2:56443
WARNING: no policy specified for kudu/127.28.6.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab.
Entry for principal kudu/127.28.6.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab.
I20260810 00:50:15.112259 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.3:0
--local_ip_for_outbound_sockets=127.28.6.3
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:39939
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:40671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:50:15.457227 26297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:50:15.457525 26297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:50:15.457708 26297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:50:15.464562 26297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:50:15.464694 26297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:50:15.464879 26297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.3
I20260810 00:50:15.474289 26297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:40671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.3.keytab
--principal=kudu/127.28.6.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:39939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.3
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:50:15.475510 26297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:50:15.476815 26297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:50:15.489966 26303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:15.489809 26304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:15.490110 26306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:50:15.490698 26297 server_base.cc:1034] running on GCE node
I20260810 00:50:15.491390 26297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:50:15.492314 26297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:50:15.493544 26297 hybrid_clock.cc:648] HybridClock initialized: now 1786323015493499 us; error 56 us; skew 500 ppm
I20260810 00:50:15.493857 26297 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:50:15 dist-test-slave-1nl6 krb5kdc[25701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323015, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:50:15.496942 26297 init.cc:377] Logged in from keytab as kudu/127.28.6.3@KRBTEST.COM (short username kudu)
I20260810 00:50:15.498421 26297 webserver.cc:460] Webserver started at http://127.28.6.3:37679/ using document root <none> and password file <none>
I20260810 00:50:15.499174 26297 fs_manager.cc:359] Metadata directory not provided
I20260810 00:50:15.499329 26297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:50:15.499639 26297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:50:15.502403 26297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-2/data/instance:
uuid: "1e4a524f8f8c471ebad8e1f0346a5496"
format_stamp: "Formatted at 2026-08-10 00:50:15 on dist-test-slave-1nl6"
server_key: "39d70e6d965829d8ebea642923c58b0e"
server_key_iv: "f594ae6b04403c7c07aea7430da54883"
server_key_version: "encryptionkey@0"
I20260810 00:50:15.503216 26297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance:
uuid: "1e4a524f8f8c471ebad8e1f0346a5496"
format_stamp: "Formatted at 2026-08-10 00:50:15 on dist-test-slave-1nl6"
server_key: "39d70e6d965829d8ebea642923c58b0e"
server_key_iv: "f594ae6b04403c7c07aea7430da54883"
server_key_version: "encryptionkey@0"
I20260810 00:50:15.508504 26297 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260810 00:50:15.512677 26318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:15.513885 26297 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 00:50:15.514056 26297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-2/wal
uuid: "1e4a524f8f8c471ebad8e1f0346a5496"
format_stamp: "Formatted at 2026-08-10 00:50:15 on dist-test-slave-1nl6"
server_key: "39d70e6d965829d8ebea642923c58b0e"
server_key_iv: "f594ae6b04403c7c07aea7430da54883"
server_key_version: "encryptionkey@0"
I20260810 00:50:15.514246 26297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:50:15.527938 26297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:50:15.532112 26297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:50:15.532362 26297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:50:15.533192 26297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:50:15.534399 26297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:50:15.534472 26297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:15.534552 26297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:50:15.534597 26297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:15.541955 26076 heartbeater.cc:503] Master 127.28.6.62:39939 was elected leader, sending a full tablet report...
I20260810 00:50:15.567490 26297 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.3:39871
I20260810 00:50:15.567576 26494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.3:39871 every 8 connection(s)
I20260810 00:50:15.568702 26297 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
I20260810 00:50:15.569646 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 26297
I20260810 00:50:15.569861 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance
I20260810 00:50:15.570389 28696 external_mini_cluster.cc:1428] Setting key 13fd2447bc7203f2c1c04e0309efa124
Aug 10 00:50:15 dist-test-slave-1nl6 krb5kdc[25701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323015, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.3@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:50:15.588582 26498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:15.571383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:35899 (local address 127.28.6.62:39939)
0810 00:50:15.571841 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:15.571847 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:15.573065 (+ 1218us) server_negotiation.cc:407] Connection header received
0810 00:50:15.574882 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:15.574885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:15.575017 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:15.575107 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:15.576637 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:15.577491 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:15.579015 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:15.579405 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:15.579496 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:15.582838 (+ 3342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:15.582843 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:15.582846 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:50:15.582876 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:15.586050 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:15.586695 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:15.586699 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:15.586701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:15.586820 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:15.587240 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:15.587243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:15.587245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:15.587485 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:15.587593 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:15.588170 (+ 577us) server_negotiation.cc:299] Negotiation successful
0810 00:50:15.588327 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":228,"threads_started":1}
I20260810 00:50:15.589455 26499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:15.572042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:39939 (local address 127.28.6.3:35899)
0810 00:50:15.572891 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:15.572946 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:15.574645 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:15.575313 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:15.575320 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:15.575860 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:50:15.576456 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:15.576476 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:15.577628 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:15.577632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:15.578819 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:15.578837 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:15.579591 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:15.579596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:15.579829 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:15.580452 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:50:15.580473 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:50:15.582659 (+ 2186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:50:15.586173 (+ 3514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:50:15.586179 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:50:15.586190 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:50:15.586596 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:50:15.586949 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:50:15.586954 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:50:15.586957 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:50:15.587140 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:50:15.587592 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:50:15.587597 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:50:15.587863 (+ 266us) client_negotiation.cc:769] Sending connection context
0810 00:50:15.588084 (+ 221us) client_negotiation.cc:240] Negotiation successful
0810 00:50:15.588425 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":354,"threads_started":1}
I20260810 00:50:15.590631 26496 heartbeater.cc:347] Connected to a master server at 127.28.6.62:39939
I20260810 00:50:15.590983 26496 heartbeater.cc:464] Registering TS with master...
I20260810 00:50:15.591694 26496 heartbeater.cc:511] Master 127.28.6.62:39939 requested a full tablet report, sending...
I20260810 00:50:15.593058 25777 ts_manager.cc:194] Registered new tserver with Master: 1e4a524f8f8c471ebad8e1f0346a5496 (127.28.6.3:39871)
I20260810 00:50:15.593839 25777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.6.3@KRBTEST.COM'} at 127.28.6.3:35899
I20260810 00:50:15.597692 28696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 00:50:15.613806 26498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:15.601770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34816 (local address 127.28.6.62:39939)
0810 00:50:15.601916 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:15.601920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:15.602366 (+ 446us) server_negotiation.cc:407] Connection header received
0810 00:50:15.602495 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:15.602498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:15.602563 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:15.602639 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:15.604101 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:15.604751 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:15.605963 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:15.606341 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:15.606432 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:15.608887 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:15.608892 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:15.608896 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:50:15.608923 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:15.611265 (+ 2342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:15.611855 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:15.611858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:15.611861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:15.611963 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:15.612369 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:15.612374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:15.612377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:15.612679 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:15.612952 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:15.613294 (+ 342us) server_negotiation.cc:299] Negotiation successful
0810 00:50:15.613506 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 00:50:15.617262 25777 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 00:50:15.624280 25777 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 00:50:15.648663 26521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:15.636443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53322 (local address 127.28.6.1:33997)
0810 00:50:15.638691 (+ 2248us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:15.638703 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:15.639167 (+ 464us) server_negotiation.cc:407] Connection header received
0810 00:50:15.639280 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:15.639287 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:15.639547 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:15.640982 (+ 1435us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:50:15.641228 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:15.642993 (+ 1765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:15.645903 (+ 2910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:15.647449 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:15.647599 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:15.647837 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:15.648152 (+ 315us) server_negotiation.cc:299] Negotiation successful
0810 00:50:15.648338 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2025,"thread_start_us":1862,"threads_started":1}
I20260810 00:50:15.648715 26523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:15.636415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:33997 (local address 127.0.0.1:53322)
0810 00:50:15.638994 (+ 2579us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:15.639015 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:15.639134 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:15.639788 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:15.639796 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:15.639847 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:50:15.640358 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:15.640373 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:15.643145 (+ 2772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:15.643149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:15.645677 (+ 2528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:15.645691 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:15.647674 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:15.647680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:15.647893 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:15.647948 (+ 55us) client_negotiation.cc:769] Sending connection context
0810 00:50:15.648079 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 00:50:15.648260 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2421,"mutex_wait_us":75,"spinlock_wait_cycles":5376,"thread_start_us":2274,"threads_started":1}
I20260810 00:50:15.650077 26519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:15.635963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:39871 (local address 127.0.0.1:36544)
0810 00:50:15.636867 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:15.636926 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:15.637137 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:15.639788 (+ 2651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:15.639796 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:15.639847 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:50:15.640784 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:15.640796 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:15.643415 (+ 2619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:15.643420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:15.646611 (+ 3191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:15.646626 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:15.649198 (+ 2572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:15.649206 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:15.649389 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:15.649425 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 00:50:15.649561 (+ 136us) client_negotiation.cc:240] Negotiation successful
0810 00:50:15.649685 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":338,"spinlock_wait_cycles":4608,"thread_start_us":629,"threads_started":1}
I20260810 00:50:15.650094 26520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:15.636219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36544 (local address 127.28.6.3:39871)
0810 00:50:15.639242 (+ 3023us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:15.639250 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:15.639278 (+ 28us) server_negotiation.cc:407] Connection header received
0810 00:50:15.639358 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:15.639364 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:15.639545 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:15.641293 (+ 1748us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:50:15.641587 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:15.643254 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:15.646815 (+ 3561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:15.648202 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:15.649207 (+ 1005us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:15.649432 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:15.649606 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 00:50:15.649758 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2864,"thread_start_us":2705,"threads_started":1}
I20260810 00:50:15.652947 25976 tablet_service.cc:1467] Processing CreateTablet for tablet dd4afb2d9894423bb9692c17f84aad76 (DEFAULT_TABLE table=test-table [id=0f245d6f268544d3b9e2c05fc05ece23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:50:15.653436 26394 tablet_service.cc:1467] Processing CreateTablet for tablet dd4afb2d9894423bb9692c17f84aad76 (DEFAULT_TABLE table=test-table [id=0f245d6f268544d3b9e2c05fc05ece23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:50:15.654567 25976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd4afb2d9894423bb9692c17f84aad76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:50:15.654940 26394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd4afb2d9894423bb9692c17f84aad76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:50:15.655184 26526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:15.639110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:36115 (local address 127.0.0.1:37998)
0810 00:50:15.645988 (+ 6878us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:15.646006 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:15.646124 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:15.646772 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:15.646777 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:15.646803 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:50:15.647223 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:15.647233 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:15.650188 (+ 2955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:15.650193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:15.652226 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:15.652243 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:15.654371 (+ 2128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:15.654380 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:15.654565 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:15.654596 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 00:50:15.654690 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 00:50:15.654839 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6726,"thread_start_us":6561,"threads_started":1}
I20260810 00:50:15.655330 26527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:15.639669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37998 (local address 127.28.6.2:36115)
0810 00:50:15.646242 (+ 6573us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:15.646249 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:15.646276 (+ 27us) server_negotiation.cc:407] Connection header received
0810 00:50:15.646390 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:15.646397 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:15.646576 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:15.646788 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:50:15.647400 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:15.648976 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:15.652413 (+ 3437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:15.653900 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:15.654033 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:15.654220 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:15.654814 (+ 594us) server_negotiation.cc:299] Negotiation successful
0810 00:50:15.655021 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6383,"thread_start_us":6509,"threads_started":1}
I20260810 00:50:15.662606 26184 tablet_service.cc:1467] Processing CreateTablet for tablet dd4afb2d9894423bb9692c17f84aad76 (DEFAULT_TABLE table=test-table [id=0f245d6f268544d3b9e2c05fc05ece23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:50:15.663761 26184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd4afb2d9894423bb9692c17f84aad76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:50:15.668005 26528 tablet_bootstrap.cc:492] T dd4afb2d9894423bb9692c17f84aad76 P 1e4a524f8f8c471ebad8e1f0346a5496: Bootstrap starting.
I20260810 00:50:15.668846 26529 tablet_bootstrap.cc:492] T dd4afb2d9894423bb9692c17f84aad76 P c457caac937b4966903cb1057679ef38: Bootstrap starting.
I20260810 00:50:15.671825 26529 tablet_bootstrap.cc:654] T dd4afb2d9894423bb9692c17f84aad76 P c457caac937b4966903cb1057679ef38: Neither blocks nor log segments found. Creating new log.
I20260810 00:50:15.672401 26528 tablet_bootstrap.cc:654] T dd4afb2d9894423bb9692c17f84aad76 P 1e4a524f8f8c471ebad8e1f0346a5496: Neither blocks nor log segments found. Creating new log.
I20260810 00:50:15.672919 26529 log.cc:824] T dd4afb2d9894423bb9692c17f84aad76 P c457caac937b4966903cb1057679ef38: Log is configured to *not* fsync() on all Append() calls
I20260810 00:50:15.673669 26528 log.cc:824] T dd4afb2d9894423bb9692c17f84aad76 P 1e4a524f8f8c471ebad8e1f0346a5496: Log is configured to *not* fsync() on all Append() calls
I20260810 00:50:15.675921 26529 tablet_bootstrap.cc:492] T dd4afb2d9894423bb9692c17f84aad76 P c457caac937b4966903cb1057679ef38: No bootstrap required, opened a new log
I20260810 00:50:15.676188 26529 ts_tablet_manager.cc:1397] T dd4afb2d9894423bb9692c17f84aad76 P c457caac937b4966903cb1057679ef38: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260810 00:50:15.676628 26528 tablet_bootstrap.cc:492] T dd4afb2d9894423bb9692c17f84aad76 P 1e4a524f8f8c471ebad8e1f0346a5496: No bootstrap required, opened a new log
I20260810 00:50:15.676887 26528 ts_tablet_manager.cc:1397] T dd4afb2d9894423bb9692c17f84aad76 P 1e4a524f8f8c471ebad8e1f0346a5496: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260810 00:50:15.677680 26535 tablet_bootstrap.cc:492] T dd4afb2d9894423bb9692c17f84aad76 P 52ddf38211c84321808ce4ae4ed23dd1: Bootstrap starting.
I20260810 00:50:15.680742 26535 tablet_bootstrap.cc:654] T dd4afb2d9894423bb9692c17f84aad76 P 52ddf38211c84321808ce4ae4ed23dd1: Neither blocks nor log segments found. Creating new log.
I20260810 00:50:15.681792 26535 log.cc:824] T dd4afb2d9894423bb9692c17f84aad76 P 52ddf38211c84321808ce4ae4ed23dd1: Log is configured to *not* fsync() on all Append() calls
I20260810 00:50:15.684556 26535 tablet_bootstrap.cc:492] T dd4afb2d9894423bb9692c17f84aad76 P 52ddf38211c84321808ce4ae4ed23dd1: No bootstrap required, opened a new log
I20260810 00:50:15.684816 26535 ts_tablet_manager.cc:1397] T dd4afb2d9894423bb9692c17f84aad76 P 52ddf38211c84321808ce4ae4ed23dd1: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 00:50:15.686977 26529 raft_consensus.cc:348] T dd4afb2d9894423bb9692c17f84aad76 P c457caac937b4966903cb1057679ef38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4a524f8f8c471ebad8e1f0346a5496" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39871 } } peers { permanent_uuid: "c457caac937b4966903cb1057679ef38" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 33997 } } peers { permanent_uuid: "52ddf38211c84321808ce4ae4ed23dd1" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 36115 } }
I20260810 00:50:15.687410 26529 raft_consensus.cc:374] T dd4afb2d9894423bb9692c17f84aad76 P c457caac937b4966903cb1057679ef38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:50:15.687521 26529 raft_consensus.cc:729] T dd4afb2d9894423bb9692c17f84aad76 P c457caac937b4966903cb1057679ef38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c457caac937b4966903cb1057679ef38, State: Initialized, Role: FOLLOWER
I20260810 00:50:15.688164 26529 consensus_queue.cc:260] T dd4afb2d9894423bb9692c17f84aad76 P c457caac937b4966903cb1057679ef38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4a524f8f8c471ebad8e1f0346a5496" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39871 } } peers { permanent_uuid: "c457caac937b4966903cb1057679ef38" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 33997 } } peers { permanent_uuid: "52ddf38211c84321808ce4ae4ed23dd1" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 36115 } }
I20260810 00:50:15.688542 26528 raft_consensus.cc:348] T dd4afb2d9894423bb9692c17f84aad76 P 1e4a524f8f8c471ebad8e1f0346a5496 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4a524f8f8c471ebad8e1f0346a5496" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39871 } } peers { permanent_uuid: "c457caac937b4966903cb1057679ef38" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 33997 } } peers { permanent_uuid: "52ddf38211c84321808ce4ae4ed23dd1" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 36115 } }
I20260810 00:50:15.688984 26528 raft_consensus.cc:374] T dd4afb2d9894423bb9692c17f84aad76 P 1e4a524f8f8c471ebad8e1f0346a5496 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:50:15.689090 26528 raft_consensus.cc:729] T dd4afb2d9894423bb9692c17f84aad76 P 1e4a524f8f8c471ebad8e1f0346a5496 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e4a524f8f8c471ebad8e1f0346a5496, State: Initialized, Role: FOLLOWER
I20260810 00:50:15.689646 26528 consensus_queue.cc:260] T dd4afb2d9894423bb9692c17f84aad76 P 1e4a524f8f8c471ebad8e1f0346a5496 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4a524f8f8c471ebad8e1f0346a5496" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39871 } } peers { permanent_uuid: "c457caac937b4966903cb1057679ef38" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 33997 } } peers { permanent_uuid: "52ddf38211c84321808ce4ae4ed23dd1" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 36115 } }
I20260810 00:50:15.690007 26529 ts_tablet_manager.cc:1428] T dd4afb2d9894423bb9692c17f84aad76 P c457caac937b4966903cb1057679ef38: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260810 00:50:15.691990 26528 ts_tablet_manager.cc:1428] T dd4afb2d9894423bb9692c17f84aad76 P 1e4a524f8f8c471ebad8e1f0346a5496: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260810 00:50:15.692373 26496 heartbeater.cc:503] Master 127.28.6.62:39939 was elected leader, sending a full tablet report...
I20260810 00:50:15.696096 26535 raft_consensus.cc:348] T dd4afb2d9894423bb9692c17f84aad76 P 52ddf38211c84321808ce4ae4ed23dd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4a524f8f8c471ebad8e1f0346a5496" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39871 } } peers { permanent_uuid: "c457caac937b4966903cb1057679ef38" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 33997 } } peers { permanent_uuid: "52ddf38211c84321808ce4ae4ed23dd1" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 36115 } }
I20260810 00:50:15.696424 26535 raft_consensus.cc:374] T dd4afb2d9894423bb9692c17f84aad76 P 52ddf38211c84321808ce4ae4ed23dd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:50:15.696547 26535 raft_consensus.cc:729] T dd4afb2d9894423bb9692c17f84aad76 P 52ddf38211c84321808ce4ae4ed23dd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ddf38211c84321808ce4ae4ed23dd1, State: Initialized, Role: FOLLOWER
I20260810 00:50:15.697121 26535 consensus_queue.cc:260] T dd4afb2d9894423bb9692c17f84aad76 P 52ddf38211c84321808ce4ae4ed23dd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4a524f8f8c471ebad8e1f0346a5496" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39871 } } peers { permanent_uuid: "c457caac937b4966903cb1057679ef38" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 33997 } } peers { permanent_uuid: "52ddf38211c84321808ce4ae4ed23dd1" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 36115 } }
I20260810 00:50:15.698073 26290 heartbeater.cc:503] Master 127.28.6.62:39939 was elected leader, sending a full tablet report...
I20260810 00:50:15.698563 26535 ts_tablet_manager.cc:1428] T dd4afb2d9894423bb9692c17f84aad76 P 52ddf38211c84321808ce4ae4ed23dd1: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260810 00:50:15.715504 26540 raft_consensus.cc:482] T dd4afb2d9894423bb9692c17f84aad76 P c457caac937b4966903cb1057679ef38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:50:15.715806 26540 raft_consensus.cc:504] T dd4afb2d9894423bb9692c17f84aad76 P c457caac937b4966903cb1057679ef38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4a524f8f8c471ebad8e1f0346a5496" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39871 } } peers { permanent_uuid: "c457caac937b4966903cb1057679ef38" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 33997 } } peers { permanent_uuid: "52ddf38211c84321808ce4ae4ed23dd1" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 36115 } }
I20260810 00:50:15.717519 26540 leader_election.cc:288] T dd4afb2d9894423bb9692c17f84aad76 P c457caac937b4966903cb1057679ef38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e4a524f8f8c471ebad8e1f0346a5496 (127.28.6.3:39871), 52ddf38211c84321808ce4ae4ed23dd1 (127.28.6.2:36115)
I20260810 00:50:15.724488 26527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:15.717943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:58811 (local address 127.28.6.2:36115)
0810 00:50:15.718180 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:15.718189 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:15.718855 (+ 666us) server_negotiation.cc:407] Connection header received
0810 00:50:15.719107 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:15.719112 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:15.719172 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:15.719278 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:50:15.719947 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:15.720726 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:15.722564 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:15.723390 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:15.723545 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:15.723648 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:15.724095 (+ 447us) server_negotiation.cc:299] Negotiation successful
0810 00:50:15.724215 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 00:50:15.724502 26520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:15.717808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:58631 (local address 127.28.6.3:39871)
0810 00:50:15.718022 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:15.718030 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:15.718885 (+ 855us) server_negotiation.cc:407] Connection header received
0810 00:50:15.719117 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:15.719128 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:15.719189 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:15.719305 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:50:15.720080 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:15.720810 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:15.722620 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:15.723457 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:15.723607 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:15.723719 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:15.724121 (+ 402us) server_negotiation.cc:299] Negotiation successful
0810 00:50:15.724248 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 00:50:15.724658 26545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:15.717856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:39871 (local address 127.28.6.1:58631)
0810 00:50:15.718767 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:15.718782 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:15.718968 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:15.719443 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:15.719448 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:15.719489 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:50:15.719918 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:15.719928 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:15.720969 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:15.720974 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:15.722458 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:15.722471 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:15.723685 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:15.723691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:15.723961 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:15.723985 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 00:50:15.724092 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 00:50:15.724212 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"mutex_wait_us":77,"thread_start_us":564,"threads_started":1}
I20260810 00:50:15.725113 26544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:15.717772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:36115 (local address 127.28.6.1:58811)
0810 00:50:15.718726 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:15.718748 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:15.718937 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:15.719306 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:15.719311 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:15.719337 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:50:15.719793 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:15.719803 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:15.720853 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:15.720861 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:15.722402 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:15.722418 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:15.723622 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:15.723628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:15.723936 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:15.723962 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 00:50:15.724067 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 00:50:15.724192 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"mutex_wait_us":85,"thread_start_us":531,"threads_started":1}
I20260810 00:50:15.725713 26430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd4afb2d9894423bb9692c17f84aad76" candidate_uuid: "c457caac937b4966903cb1057679ef38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e4a524f8f8c471ebad8e1f0346a5496" is_pre_election: true
I20260810 00:50:15.726223 26430 raft_consensus.cc:2454] T dd4afb2d9894423bb9692c17f84aad76 P 1e4a524f8f8c471ebad8e1f0346a5496 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c457caac937b4966903cb1057679ef38 in term 0.
I20260810 00:50:15.726866 25899 leader_election.cc:302] T dd4afb2d9894423bb9692c17f84aad76 P c457caac937b4966903cb1057679ef38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e4a524f8f8c471ebad8e1f0346a5496, c457caac937b4966903cb1057679ef38; no voters:
I20260810 00:50:15.727336 26540 raft_consensus.cc:2781] T dd4afb2d9894423bb9692c17f84aad76 P c457caac937b4966903cb1057679ef38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 00:50:15.727510 26540 raft_consensus.cc:482] T dd4afb2d9894423bb9692c17f84aad76 P c457caac937b4966903cb1057679ef38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 00:50:15.727619 26540 raft_consensus.cc:3037] T dd4afb2d9894423bb9692c17f84aad76 P c457caac937b4966903cb1057679ef38 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:50:15.727495 26211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd4afb2d9894423bb9692c17f84aad76" candidate_uuid: "c457caac937b4966903cb1057679ef38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ddf38211c84321808ce4ae4ed23dd1" is_pre_election: true
I20260810 00:50:15.727994 26211 raft_consensus.cc:2454] T dd4afb2d9894423bb9692c17f84aad76 P 52ddf38211c84321808ce4ae4ed23dd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c457caac937b4966903cb1057679ef38 in term 0.
I20260810 00:50:15.730047 26540 raft_consensus.cc:504] T dd4afb2d9894423bb9692c17f84aad76 P c457caac937b4966903cb1057679ef38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4a524f8f8c471ebad8e1f0346a5496" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39871 } } peers { permanent_uuid: "c457caac937b4966903cb1057679ef38" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 33997 } } peers { permanent_uuid: "52ddf38211c84321808ce4ae4ed23dd1" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 36115 } }
I20260810 00:50:15.730718 26540 leader_election.cc:288] T dd4afb2d9894423bb9692c17f84aad76 P c457caac937b4966903cb1057679ef38 [CANDIDATE]: Term 1 election: Requested vote from peers 1e4a524f8f8c471ebad8e1f0346a5496 (127.28.6.3:39871), 52ddf38211c84321808ce4ae4ed23dd1 (127.28.6.2:36115)
I20260810 00:50:15.731032 26430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd4afb2d9894423bb9692c17f84aad76" candidate_uuid: "c457caac937b4966903cb1057679ef38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e4a524f8f8c471ebad8e1f0346a5496"
I20260810 00:50:15.731237 26211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd4afb2d9894423bb9692c17f84aad76" candidate_uuid: "c457caac937b4966903cb1057679ef38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ddf38211c84321808ce4ae4ed23dd1"
I20260810 00:50:15.731267 26430 raft_consensus.cc:3037] T dd4afb2d9894423bb9692c17f84aad76 P 1e4a524f8f8c471ebad8e1f0346a5496 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:50:15.731436 26211 raft_consensus.cc:3037] T dd4afb2d9894423bb9692c17f84aad76 P 52ddf38211c84321808ce4ae4ed23dd1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:50:15.734568 26430 raft_consensus.cc:2454] T dd4afb2d9894423bb9692c17f84aad76 P 1e4a524f8f8c471ebad8e1f0346a5496 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c457caac937b4966903cb1057679ef38 in term 1.
I20260810 00:50:15.734668 26211 raft_consensus.cc:2454] T dd4afb2d9894423bb9692c17f84aad76 P 52ddf38211c84321808ce4ae4ed23dd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c457caac937b4966903cb1057679ef38 in term 1.
I20260810 00:50:15.735001 25899 leader_election.cc:302] T dd4afb2d9894423bb9692c17f84aad76 P c457caac937b4966903cb1057679ef38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e4a524f8f8c471ebad8e1f0346a5496, c457caac937b4966903cb1057679ef38; no voters:
I20260810 00:50:15.735306 26540 raft_consensus.cc:2781] T dd4afb2d9894423bb9692c17f84aad76 P c457caac937b4966903cb1057679ef38 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:50:15.735766 26540 raft_consensus.cc:686] T dd4afb2d9894423bb9692c17f84aad76 P c457caac937b4966903cb1057679ef38 [term 1 LEADER]: Becoming Leader. State: Replica: c457caac937b4966903cb1057679ef38, State: Running, Role: LEADER
I20260810 00:50:15.736236 26540 consensus_queue.cc:237] T dd4afb2d9894423bb9692c17f84aad76 P c457caac937b4966903cb1057679ef38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4a524f8f8c471ebad8e1f0346a5496" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39871 } } peers { permanent_uuid: "c457caac937b4966903cb1057679ef38" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 33997 } } peers { permanent_uuid: "52ddf38211c84321808ce4ae4ed23dd1" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 36115 } }
I20260810 00:50:15.740693 25777 catalog_manager.cc:5495] T dd4afb2d9894423bb9692c17f84aad76 P c457caac937b4966903cb1057679ef38 reported cstate change: term changed from 0 to 1, leader changed from <none> to c457caac937b4966903cb1057679ef38 (127.28.6.1). New cstate: current_term: 1 leader_uuid: "c457caac937b4966903cb1057679ef38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e4a524f8f8c471ebad8e1f0346a5496" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c457caac937b4966903cb1057679ef38" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 33997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ddf38211c84321808ce4ae4ed23dd1" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 36115 } health_report { overall_health: UNKNOWN } } }
W20260810 00:50:15.760644 26080 tablet.cc:2369] T dd4afb2d9894423bb9692c17f84aad76 P c457caac937b4966903cb1057679ef38: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:50:15.769919 26521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:15.765034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53338 (local address 127.28.6.1:33997)
0810 00:50:15.765197 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:15.765201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:15.765219 (+ 18us) server_negotiation.cc:407] Connection header received
0810 00:50:15.765398 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:15.765402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:15.765491 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:15.765610 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:50:15.765938 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:15.766671 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:15.768055 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:15.768444 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:15.768539 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:15.768983 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:50:15.769458 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:50:15.769566 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:15.769642 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 00:50:15.769702 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 00:50:15.777851 26430 raft_consensus.cc:1261] T dd4afb2d9894423bb9692c17f84aad76 P 1e4a524f8f8c471ebad8e1f0346a5496 [term 1 FOLLOWER]: Refusing update from remote peer c457caac937b4966903cb1057679ef38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:50:15.777854 26211 raft_consensus.cc:1261] T dd4afb2d9894423bb9692c17f84aad76 P 52ddf38211c84321808ce4ae4ed23dd1 [term 1 FOLLOWER]: Refusing update from remote peer c457caac937b4966903cb1057679ef38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:50:15.778812 26546 consensus_queue.cc:1035] T dd4afb2d9894423bb9692c17f84aad76 P c457caac937b4966903cb1057679ef38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e4a524f8f8c471ebad8e1f0346a5496" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 39871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:50:15.779186 26540 consensus_queue.cc:1035] T dd4afb2d9894423bb9692c17f84aad76 P c457caac937b4966903cb1057679ef38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ddf38211c84321808ce4ae4ed23dd1" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 36115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 00:50:15.787851 26291 tablet.cc:2369] T dd4afb2d9894423bb9692c17f84aad76 P 52ddf38211c84321808ce4ae4ed23dd1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:50:15.793120 26554 mvcc.cc:205] Tried to move back new op lower bound from 7316779072616091648 to 7316779072458977280. Current Snapshot: MvccSnapshot[applied={T|T < 7316779072616091648}]
I20260810 00:50:15.804839 25777 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:50:15.805066 25777 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:50:15.808584 25777 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 729b5a3ca8b04c4aa956a4099c02253b: Sending DeleteTablet for 3 replicas of tablet dd4afb2d9894423bb9692c17f84aad76
I20260810 00:50:15.809648 26394 tablet_service.cc:1514] Processing DeleteTablet for tablet dd4afb2d9894423bb9692c17f84aad76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:50:15 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:36544
I20260810 00:50:15.809890 26184 tablet_service.cc:1514] Processing DeleteTablet for tablet dd4afb2d9894423bb9692c17f84aad76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:50:15 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:37998
I20260810 00:50:15.810743 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 25865
I20260810 00:50:15.810774 25976 tablet_service.cc:1514] Processing DeleteTablet for tablet dd4afb2d9894423bb9692c17f84aad76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:50:15 UTC) from {username='kudu', principal='kudu/127.28.6.62@KRBTEST.COM'} at 127.0.0.1:53322
I20260810 00:50:15.813673 26568 tablet_replica.cc:330] T dd4afb2d9894423bb9692c17f84aad76 P 52ddf38211c84321808ce4ae4ed23dd1: stopping tablet replica
I20260810 00:50:15.814136 26568 raft_consensus.cc:2229] T dd4afb2d9894423bb9692c17f84aad76 P 52ddf38211c84321808ce4ae4ed23dd1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:50:15.814682 26568 raft_consensus.cc:2258] T dd4afb2d9894423bb9692c17f84aad76 P 52ddf38211c84321808ce4ae4ed23dd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:50:15.817072 26568 ts_tablet_manager.cc:1905] T dd4afb2d9894423bb9692c17f84aad76 P 52ddf38211c84321808ce4ae4ed23dd1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:50:15.824749 26569 tablet_replica.cc:330] T dd4afb2d9894423bb9692c17f84aad76 P c457caac937b4966903cb1057679ef38: stopping tablet replica
I20260810 00:50:15.825347 26569 raft_consensus.cc:2229] T dd4afb2d9894423bb9692c17f84aad76 P c457caac937b4966903cb1057679ef38 [term 1 LEADER]: Raft consensus shutting down.
I20260810 00:50:15.825626 26568 ts_tablet_manager.cc:1918] T dd4afb2d9894423bb9692c17f84aad76 P 52ddf38211c84321808ce4ae4ed23dd1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:50:15.825804 26568 log.cc:1196] T dd4afb2d9894423bb9692c17f84aad76 P 52ddf38211c84321808ce4ae4ed23dd1: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-1/wal/wals/dd4afb2d9894423bb9692c17f84aad76
I20260810 00:50:15.826330 26569 raft_consensus.cc:2258] T dd4afb2d9894423bb9692c17f84aad76 P c457caac937b4966903cb1057679ef38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:50:15.826411 26568 ts_tablet_manager.cc:1939] T dd4afb2d9894423bb9692c17f84aad76 P 52ddf38211c84321808ce4ae4ed23dd1: Deleting consensus metadata
I20260810 00:50:15.828311 25758 catalog_manager.cc:4841] TS 52ddf38211c84321808ce4ae4ed23dd1 (127.28.6.2:36115): tablet dd4afb2d9894423bb9692c17f84aad76 (table test-table [id=0f245d6f268544d3b9e2c05fc05ece23]) successfully deleted
I20260810 00:50:15.828624 26569 ts_tablet_manager.cc:1905] T dd4afb2d9894423bb9692c17f84aad76 P c457caac937b4966903cb1057679ef38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:50:15.829799 26565 tablet_replica.cc:330] T dd4afb2d9894423bb9692c17f84aad76 P 1e4a524f8f8c471ebad8e1f0346a5496: stopping tablet replica
I20260810 00:50:15.830188 26565 raft_consensus.cc:2229] T dd4afb2d9894423bb9692c17f84aad76 P 1e4a524f8f8c471ebad8e1f0346a5496 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:50:15.830717 26565 raft_consensus.cc:2258] T dd4afb2d9894423bb9692c17f84aad76 P 1e4a524f8f8c471ebad8e1f0346a5496 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:50:15.832715 26565 ts_tablet_manager.cc:1905] T dd4afb2d9894423bb9692c17f84aad76 P 1e4a524f8f8c471ebad8e1f0346a5496: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:50:15.836606 26569 ts_tablet_manager.cc:1918] T dd4afb2d9894423bb9692c17f84aad76 P c457caac937b4966903cb1057679ef38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:50:15.836881 26569 log.cc:1196] T dd4afb2d9894423bb9692c17f84aad76 P c457caac937b4966903cb1057679ef38: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-0/wal/wals/dd4afb2d9894423bb9692c17f84aad76
I20260810 00:50:15.837522 26569 ts_tablet_manager.cc:1939] T dd4afb2d9894423bb9692c17f84aad76 P c457caac937b4966903cb1057679ef38: Deleting consensus metadata
I20260810 00:50:15.839334 25758 catalog_manager.cc:4841] TS c457caac937b4966903cb1057679ef38 (127.28.6.1:33997): tablet dd4afb2d9894423bb9692c17f84aad76 (table test-table [id=0f245d6f268544d3b9e2c05fc05ece23]) successfully deleted
I20260810 00:50:15.840209 26565 ts_tablet_manager.cc:1918] T dd4afb2d9894423bb9692c17f84aad76 P 1e4a524f8f8c471ebad8e1f0346a5496: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:50:15.840385 26565 log.cc:1196] T dd4afb2d9894423bb9692c17f84aad76 P 1e4a524f8f8c471ebad8e1f0346a5496: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786322687226232-28696-0/minicluster-data/ts-2/wal/wals/dd4afb2d9894423bb9692c17f84aad76
I20260810 00:50:15.840842 26565 ts_tablet_manager.cc:1939] T dd4afb2d9894423bb9692c17f84aad76 P 1e4a524f8f8c471ebad8e1f0346a5496: Deleting consensus metadata
Aug 10 00:50:15 dist-test-slave-1nl6 krb5kdc[25701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.1@KRBTEST.COM
I20260810 00:50:15.842625 25759 catalog_manager.cc:4841] TS 1e4a524f8f8c471ebad8e1f0346a5496 (127.28.6.3:39871): tablet dd4afb2d9894423bb9692c17f84aad76 (table test-table [id=0f245d6f268544d3b9e2c05fc05ece23]) successfully deleted
I20260810 00:50:15.848487 26521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:15.812011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53342 (local address 127.28.6.1:33997)
0810 00:50:15.812195 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:15.812201 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:15.829177 (+ 16976us) server_negotiation.cc:407] Connection header received
0810 00:50:15.829388 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:15.829396 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:15.829490 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:15.829600 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:15.831507 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:15.832787 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:15.835660 (+ 2873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:15.836365 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:15.836464 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:15.842248 (+ 5784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:15.842262 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:15.842278 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:50:15.842323 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:15.845577 (+ 3254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:15.846273 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:15.846288 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:15.846290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:15.846425 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:15.846829 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:15.846832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:15.846835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:15.847285 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:15.847586 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:15.847913 (+ 327us) server_negotiation.cc:299] Negotiation successful
0810 00:50:15.848158 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 00:50:15.849231 26071 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:50:15.949762 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 25865
I20260810 00:50:15.978173 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 26088
Aug 10 00:50:15 dist-test-slave-1nl6 krb5kdc[25701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.2@KRBTEST.COM
I20260810 00:50:15.993628 26527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:15.979003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38012 (local address 127.28.6.2:36115)
0810 00:50:15.979238 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:15.979243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:15.979414 (+ 171us) server_negotiation.cc:407] Connection header received
0810 00:50:15.979573 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:15.979577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:15.979654 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:15.979740 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:15.981156 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:15.981887 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:15.983370 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:15.983743 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:15.983840 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:15.988048 (+ 4208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:15.988061 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:15.988078 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:50:15.988118 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:15.991111 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:15.991756 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:15.991763 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:15.991766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:15.991908 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:15.992228 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:15.992231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:15.992233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:15.992604 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:15.992820 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:15.993093 (+ 273us) server_negotiation.cc:299] Negotiation successful
0810 00:50:15.993250 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 00:50:15.994292 26283 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:50:16.083329 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 26088
I20260810 00:50:16.106649 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 26297
Aug 10 00:50:16 dist-test-slave-1nl6 krb5kdc[25701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.3@KRBTEST.COM
I20260810 00:50:16.122488 26520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:16.107484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36552 (local address 127.28.6.3:39871)
0810 00:50:16.107680 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:16.107686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:16.107707 (+ 21us) server_negotiation.cc:407] Connection header received
0810 00:50:16.107925 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:16.107929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:16.108000 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:16.108124 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:16.109482 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:16.110223 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:16.111703 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:16.112104 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:16.112242 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:16.116776 (+ 4534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:16.116794 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:16.116813 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:50:16.116852 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:16.119609 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:16.120341 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:16.120350 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:16.120353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:16.120496 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:16.120899 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:16.120905 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:16.120912 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:16.121278 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:16.121602 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:16.121933 (+ 331us) server_negotiation.cc:299] Negotiation successful
0810 00:50:16.122180 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 00:50:16.123176 26489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:50:16.213944 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 26297
I20260810 00:50:16.235216 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 25726
I20260810 00:50:16.236299 25819 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:50:16.344403 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 25726
2026-08-10T00:50:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4496 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-10T00:50:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:50:16Z Disabled control of system clock
I20260810 00:50:16.442289 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:40399
--webserver_interface=127.28.6.62
--webserver_port=0
--builtin_ntp_servers=127.28.6.20:43469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:40399
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 00:50:16.781634 26580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:50:16.781903 26580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:50:16.782001 26580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:50:16.789011 26580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:50:16.789111 26580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:50:16.789149 26580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:50:16.789207 26580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:50:16.789247 26580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:50:16.797968 26580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:43469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:40399
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:40399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:50:16.799099 26580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:50:16.800818 26580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:50:16.812947 26587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:16.812989 26590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:16.813647 26586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:50:16.813939 26580 server_base.cc:1034] running on GCE node
I20260810 00:50:16.814688 26580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:50:16.816045 26580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:50:16.817257 26580 hybrid_clock.cc:648] HybridClock initialized: now 1786323016817208 us; error 43 us; skew 500 ppm
I20260810 00:50:16.817631 26580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 00:50:16.819926 26580 webserver.cc:460] Webserver started at http://127.28.6.62:39043/ using document root <none> and password file <none>
I20260810 00:50:16.820583 26580 fs_manager.cc:359] Metadata directory not provided
I20260810 00:50:16.820708 26580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:50:16.821050 26580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:50:16.823951 26580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "c113b05ce77e498ebd5d30f8c17fcb93"
format_stamp: "Formatted at 2026-08-10 00:50:16 on dist-test-slave-1nl6"
server_key: "f56295786e254fccd6c23a815f8fa904"
server_key_iv: "1c503a5fdc4a14558e0b1b713f53f8e4"
server_key_version: "encryptionkey@0"
I20260810 00:50:16.824754 26580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "c113b05ce77e498ebd5d30f8c17fcb93"
format_stamp: "Formatted at 2026-08-10 00:50:16 on dist-test-slave-1nl6"
server_key: "f56295786e254fccd6c23a815f8fa904"
server_key_iv: "1c503a5fdc4a14558e0b1b713f53f8e4"
server_key_version: "encryptionkey@0"
I20260810 00:50:16.829608 26580 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260810 00:50:16.832882 26601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:16.834139 26580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 00:50:16.834332 26580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "c113b05ce77e498ebd5d30f8c17fcb93"
format_stamp: "Formatted at 2026-08-10 00:50:16 on dist-test-slave-1nl6"
server_key: "f56295786e254fccd6c23a815f8fa904"
server_key_iv: "1c503a5fdc4a14558e0b1b713f53f8e4"
server_key_version: "encryptionkey@0"
I20260810 00:50:16.834539 26580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:50:16.851859 26580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:50:16.852634 26580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:50:16.852912 26580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:50:16.871430 26580 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:40399
I20260810 00:50:16.871466 26685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:40399 every 8 connection(s)
I20260810 00:50:16.872747 26580 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:50:16.876292 26686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:50:16.878008 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 26580
I20260810 00:50:16.878219 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:50:16.878805 28696 external_mini_cluster.cc:1428] Setting key df48bf52440f65e6fce810ab75a5832e
I20260810 00:50:16.889634 26686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c113b05ce77e498ebd5d30f8c17fcb93: Bootstrap starting.
I20260810 00:50:16.891906 26691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:16.880854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56652 (local address 127.28.6.62:40399)
0810 00:50:16.881848 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:16.881858 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:16.881919 (+ 61us) server_negotiation.cc:407] Connection header received
0810 00:50:16.883551 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:16.883576 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:16.884024 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:16.884401 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:16.884959 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:16.886171 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:16.887654 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:16.888298 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:16.888520 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:16.889744 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:16.889757 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:16.889776 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:16.890294 (+ 518us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:16.890366 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:16.890387 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:16.890511 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:16.890732 (+ 221us) server_negotiation.cc:299] Negotiation successful
0810 00:50:16.891045 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":361,"threads_started":1}
I20260810 00:50:16.892877 26686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c113b05ce77e498ebd5d30f8c17fcb93: Neither blocks nor log segments found. Creating new log.
I20260810 00:50:16.894223 26686 log.cc:824] T 00000000000000000000000000000000 P c113b05ce77e498ebd5d30f8c17fcb93: Log is configured to *not* fsync() on all Append() calls
I20260810 00:50:16.896907 26686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c113b05ce77e498ebd5d30f8c17fcb93: No bootstrap required, opened a new log
I20260810 00:50:16.904652 26686 raft_consensus.cc:348] T 00000000000000000000000000000000 P c113b05ce77e498ebd5d30f8c17fcb93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c113b05ce77e498ebd5d30f8c17fcb93" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 40399 } }
I20260810 00:50:16.904969 26686 raft_consensus.cc:374] T 00000000000000000000000000000000 P c113b05ce77e498ebd5d30f8c17fcb93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:50:16.905050 26686 raft_consensus.cc:729] T 00000000000000000000000000000000 P c113b05ce77e498ebd5d30f8c17fcb93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c113b05ce77e498ebd5d30f8c17fcb93, State: Initialized, Role: FOLLOWER
I20260810 00:50:16.905514 26686 consensus_queue.cc:260] T 00000000000000000000000000000000 P c113b05ce77e498ebd5d30f8c17fcb93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c113b05ce77e498ebd5d30f8c17fcb93" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 40399 } }
I20260810 00:50:16.905706 26686 raft_consensus.cc:388] T 00000000000000000000000000000000 P c113b05ce77e498ebd5d30f8c17fcb93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:50:16.905798 26686 raft_consensus.cc:482] T 00000000000000000000000000000000 P c113b05ce77e498ebd5d30f8c17fcb93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:50:16.905910 26686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c113b05ce77e498ebd5d30f8c17fcb93 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:50:16.907886 26686 raft_consensus.cc:504] T 00000000000000000000000000000000 P c113b05ce77e498ebd5d30f8c17fcb93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c113b05ce77e498ebd5d30f8c17fcb93" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 40399 } }
I20260810 00:50:16.908294 26686 leader_election.cc:302] T 00000000000000000000000000000000 P c113b05ce77e498ebd5d30f8c17fcb93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c113b05ce77e498ebd5d30f8c17fcb93; no voters:
I20260810 00:50:16.908785 26686 leader_election.cc:288] T 00000000000000000000000000000000 P c113b05ce77e498ebd5d30f8c17fcb93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:50:16.908937 26693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c113b05ce77e498ebd5d30f8c17fcb93 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:50:16.909685 26693 raft_consensus.cc:686] T 00000000000000000000000000000000 P c113b05ce77e498ebd5d30f8c17fcb93 [term 1 LEADER]: Becoming Leader. State: Replica: c113b05ce77e498ebd5d30f8c17fcb93, State: Running, Role: LEADER
I20260810 00:50:16.910256 26693 consensus_queue.cc:237] T 00000000000000000000000000000000 P c113b05ce77e498ebd5d30f8c17fcb93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c113b05ce77e498ebd5d30f8c17fcb93" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 40399 } }
I20260810 00:50:16.910831 26686 sys_catalog.cc:564] T 00000000000000000000000000000000 P c113b05ce77e498ebd5d30f8c17fcb93 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:50:16.913267 26696 sys_catalog.cc:455] T 00000000000000000000000000000000 P c113b05ce77e498ebd5d30f8c17fcb93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c113b05ce77e498ebd5d30f8c17fcb93. Latest consensus state: current_term: 1 leader_uuid: "c113b05ce77e498ebd5d30f8c17fcb93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c113b05ce77e498ebd5d30f8c17fcb93" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 40399 } } }
I20260810 00:50:16.913862 26696 sys_catalog.cc:458] T 00000000000000000000000000000000 P c113b05ce77e498ebd5d30f8c17fcb93 [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:16.913713 26695 sys_catalog.cc:455] T 00000000000000000000000000000000 P c113b05ce77e498ebd5d30f8c17fcb93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c113b05ce77e498ebd5d30f8c17fcb93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c113b05ce77e498ebd5d30f8c17fcb93" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 40399 } } }
I20260810 00:50:16.914088 26695 sys_catalog.cc:458] T 00000000000000000000000000000000 P c113b05ce77e498ebd5d30f8c17fcb93 [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:16.916100 26705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:50:16.920759 26705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:50:16.928532 26705 catalog_manager.cc:1347] Generated new cluster ID: 7be29540aa5a4a21974b3eeba0f7a9c8
I20260810 00:50:16.928651 26705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:50:16.952880 26705 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:50:16.954627 26705 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:50:16.966250 26705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c113b05ce77e498ebd5d30f8c17fcb93: Generated new TSK 0
I20260810 00:50:16.967242 26705 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 00:50:16.971331 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.1:0
--local_ip_for_outbound_sockets=127.28.6.1
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:40399
--builtin_ntp_servers=127.28.6.20:43469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 00:50:17.309733 26725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:50:17.309988 26725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:50:17.310068 26725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:50:17.317414 26725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:50:17.317502 26725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:50:17.317610 26725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.1
I20260810 00:50:17.326205 26725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:43469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.6.1
--webserver_port=0
--tserver_master_addrs=127.28.6.62:40399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.1
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:50:17.327368 26725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:50:17.328994 26725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:50:17.343432 26730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:17.343494 26734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:17.343508 26731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:50:17.344141 26725 server_base.cc:1034] running on GCE node
I20260810 00:50:17.344769 26725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:50:17.345631 26725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:50:17.346871 26725 hybrid_clock.cc:648] HybridClock initialized: now 1786323017346831 us; error 59 us; skew 500 ppm
I20260810 00:50:17.347213 26725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 00:50:17.353017 26725 webserver.cc:460] Webserver started at http://127.28.6.1:44485/ using document root <none> and password file <none>
I20260810 00:50:17.353724 26725 fs_manager.cc:359] Metadata directory not provided
I20260810 00:50:17.353860 26725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:50:17.354182 26725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:50:17.357257 26725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-0/data/instance:
uuid: "5b145e188b6647d3bdfa0484e4e317d2"
format_stamp: "Formatted at 2026-08-10 00:50:17 on dist-test-slave-1nl6"
server_key: "994b9366cbf8090b0d3f462692a2906d"
server_key_iv: "e42e6709c2057422b6ed58e70f1a6330"
server_key_version: "encryptionkey@0"
I20260810 00:50:17.358132 26725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance:
uuid: "5b145e188b6647d3bdfa0484e4e317d2"
format_stamp: "Formatted at 2026-08-10 00:50:17 on dist-test-slave-1nl6"
server_key: "994b9366cbf8090b0d3f462692a2906d"
server_key_iv: "e42e6709c2057422b6ed58e70f1a6330"
server_key_version: "encryptionkey@0"
I20260810 00:50:17.363265 26725 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260810 00:50:17.366585 26744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:17.367864 26725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 00:50:17.368046 26725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-0/wal
uuid: "5b145e188b6647d3bdfa0484e4e317d2"
format_stamp: "Formatted at 2026-08-10 00:50:17 on dist-test-slave-1nl6"
server_key: "994b9366cbf8090b0d3f462692a2906d"
server_key_iv: "e42e6709c2057422b6ed58e70f1a6330"
server_key_version: "encryptionkey@0"
I20260810 00:50:17.368273 26725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:50:17.379637 26725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:50:17.380405 26725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:50:17.380687 26725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:50:17.381628 26725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:50:17.383064 26725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:50:17.383185 26725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:17.383320 26725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:50:17.383409 26725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:17.418627 26725 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.1:38841
I20260810 00:50:17.418708 26920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.1:38841 every 8 connection(s)
I20260810 00:50:17.419891 26725 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-0/data/info.pb
I20260810 00:50:17.421691 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 26725
I20260810 00:50:17.421901 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-0/wal/instance
I20260810 00:50:17.422375 28696 external_mini_cluster.cc:1428] Setting key b361b94ce1d2232127156c0cb888ba47
I20260810 00:50:17.425628 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.2:0
--local_ip_for_outbound_sockets=127.28.6.2
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:40399
--builtin_ntp_servers=127.28.6.20:43469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 00:50:17.434386 26925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:17.422461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:49553 (local address 127.28.6.62:40399)
0810 00:50:17.422946 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:17.422952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:17.424382 (+ 1430us) server_negotiation.cc:407] Connection header received
0810 00:50:17.427081 (+ 2699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:17.427085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:17.427158 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:17.427242 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:17.428569 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:17.429266 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:17.431647 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:17.432039 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:17.432133 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:17.433679 (+ 1546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:17.433684 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:17.433687 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:17.433703 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:17.433741 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:17.433752 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:17.433821 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:17.434071 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 00:50:17.434143 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":213,"threads_started":1}
I20260810 00:50:17.435793 26928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:17.423152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:40399 (local address 127.28.6.1:49553)
0810 00:50:17.424242 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:17.424292 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:17.426754 (+ 2462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:17.427500 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:17.427513 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:17.427632 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:50:17.428402 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:17.428429 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:17.429428 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:17.429437 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:17.431509 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:17.431526 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:17.432268 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:17.432279 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:17.432600 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:17.433334 (+ 734us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 00:50:17.433339 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:50:17.433395 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 00:50:17.433489 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:50:17.433851 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:50:17.433864 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:50:17.433875 (+ 11us) client_negotiation.cc:769] Sending connection context
0810 00:50:17.434063 (+ 188us) client_negotiation.cc:240] Negotiation successful
0810 00:50:17.434446 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":315,"threads_started":1}
I20260810 00:50:17.437275 26922 heartbeater.cc:347] Connected to a master server at 127.28.6.62:40399
I20260810 00:50:17.437563 26922 heartbeater.cc:464] Registering TS with master...
I20260810 00:50:17.438419 26922 heartbeater.cc:511] Master 127.28.6.62:40399 requested a full tablet report, sending...
I20260810 00:50:17.440845 26630 ts_manager.cc:194] Registered new tserver with Master: 5b145e188b6647d3bdfa0484e4e317d2 (127.28.6.1:38841)
I20260810 00:50:17.443091 26630 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.6.1:49553
W20260810 00:50:17.777354 26929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:50:17.777634 26929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:50:17.777717 26929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:50:17.784703 26929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:50:17.784793 26929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:50:17.784893 26929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.2
I20260810 00:50:17.793503 26929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:43469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.6.2
--webserver_port=0
--tserver_master_addrs=127.28.6.62:40399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.2
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:50:17.794600 26929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:50:17.796382 26929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:50:17.809657 26941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:17.809864 26935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:17.810316 26936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:50:17.810876 26929 server_base.cc:1034] running on GCE node
I20260810 00:50:17.811486 26929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:50:17.812426 26929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:50:17.813629 26929 hybrid_clock.cc:648] HybridClock initialized: now 1786323017813590 us; error 35 us; skew 500 ppm
I20260810 00:50:17.813954 26929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 00:50:17.816262 26929 webserver.cc:460] Webserver started at http://127.28.6.2:42781/ using document root <none> and password file <none>
I20260810 00:50:17.816979 26929 fs_manager.cc:359] Metadata directory not provided
I20260810 00:50:17.817099 26929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:50:17.817444 26929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:50:17.820369 26929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-1/data/instance:
uuid: "25eda644720d4520a282b044bc54261a"
format_stamp: "Formatted at 2026-08-10 00:50:17 on dist-test-slave-1nl6"
server_key: "c2205ac9bb72ff5c98fca51019579214"
server_key_iv: "1835f3478eb64fbc66af44153106f022"
server_key_version: "encryptionkey@0"
I20260810 00:50:17.821163 26929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance:
uuid: "25eda644720d4520a282b044bc54261a"
format_stamp: "Formatted at 2026-08-10 00:50:17 on dist-test-slave-1nl6"
server_key: "c2205ac9bb72ff5c98fca51019579214"
server_key_iv: "1835f3478eb64fbc66af44153106f022"
server_key_version: "encryptionkey@0"
I20260810 00:50:17.826064 26929 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 00:50:17.829510 26951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:17.830801 26929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 00:50:17.831007 26929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-1/wal
uuid: "25eda644720d4520a282b044bc54261a"
format_stamp: "Formatted at 2026-08-10 00:50:17 on dist-test-slave-1nl6"
server_key: "c2205ac9bb72ff5c98fca51019579214"
server_key_iv: "1835f3478eb64fbc66af44153106f022"
server_key_version: "encryptionkey@0"
I20260810 00:50:17.831223 26929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:50:17.851013 26929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:50:17.851781 26929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:50:17.852070 26929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:50:17.853048 26929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:50:17.854532 26929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:50:17.854655 26929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:17.854791 26929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:50:17.854887 26929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:17.890628 26929 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.2:44161
I20260810 00:50:17.890713 27127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.2:44161 every 8 connection(s)
I20260810 00:50:17.891963 26929 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-1/data/info.pb
I20260810 00:50:17.901266 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 26929
I20260810 00:50:17.901636 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-1/wal/instance
I20260810 00:50:17.902166 28696 external_mini_cluster.cc:1428] Setting key e80a70e39158d576b2d68f3a337db83e
I20260810 00:50:17.903854 26925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:17.894464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.2:51675 (local address 127.28.6.62:40399)
0810 00:50:17.894637 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:17.894641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:17.895983 (+ 1342us) server_negotiation.cc:407] Connection header received
0810 00:50:17.897640 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:17.897644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:17.897714 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:17.897802 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:17.898755 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:17.899551 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:17.901054 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:17.901463 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:17.901604 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:17.902998 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:17.903003 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:17.903006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:17.903024 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:17.903060 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:17.903072 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:17.903176 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:17.903395 (+ 219us) server_negotiation.cc:299] Negotiation successful
0810 00:50:17.903502 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 00:50:17.905043 27133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:17.894883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:40399 (local address 127.28.6.2:51675)
0810 00:50:17.895816 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:17.895859 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:17.897412 (+ 1553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:17.897992 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:17.898000 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:17.898076 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:50:17.898611 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:17.898626 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:17.899684 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:17.899688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:17.900899 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:17.900911 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:17.901693 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:17.901701 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:17.901970 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:17.902653 (+ 683us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 00:50:17.902656 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:50:17.902693 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 00:50:17.902776 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:50:17.903176 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:50:17.903189 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:50:17.903200 (+ 11us) client_negotiation.cc:769] Sending connection context
0810 00:50:17.903380 (+ 180us) client_negotiation.cc:240] Negotiation successful
0810 00:50:17.903777 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":286,"threads_started":1}
I20260810 00:50:17.905282 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.6.3:0
--local_ip_for_outbound_sockets=127.28.6.3
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:40399
--builtin_ntp_servers=127.28.6.20:43469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 00:50:17.907912 27130 heartbeater.cc:347] Connected to a master server at 127.28.6.62:40399
I20260810 00:50:17.908223 27130 heartbeater.cc:464] Registering TS with master...
I20260810 00:50:17.909093 27130 heartbeater.cc:511] Master 127.28.6.62:40399 requested a full tablet report, sending...
I20260810 00:50:17.910609 26630 ts_manager.cc:194] Registered new tserver with Master: 25eda644720d4520a282b044bc54261a (127.28.6.2:44161)
I20260810 00:50:17.911516 26630 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.6.2:51675
W20260810 00:50:18.250882 27134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:50:18.251168 27134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:50:18.251253 27134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:50:18.258222 27134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:50:18.258312 27134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:50:18.258412 27134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.6.3
I20260810 00:50:18.267033 27134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:43469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.6.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.6.3
--webserver_port=0
--tserver_master_addrs=127.28.6.62:40399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.6.3
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:50:18.268179 27134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:50:18.269897 27134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:50:18.283061 27147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:18.283346 27143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:18.283373 27144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:50:18.283823 27134 server_base.cc:1034] running on GCE node
I20260810 00:50:18.284335 27134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:50:18.285233 27134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:50:18.286463 27134 hybrid_clock.cc:648] HybridClock initialized: now 1786323018286395 us; error 35 us; skew 500 ppm
I20260810 00:50:18.286801 27134 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 00:50:18.288981 27134 webserver.cc:460] Webserver started at http://127.28.6.3:37575/ using document root <none> and password file <none>
I20260810 00:50:18.289664 27134 fs_manager.cc:359] Metadata directory not provided
I20260810 00:50:18.289793 27134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:50:18.290131 27134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:50:18.293044 27134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-2/data/instance:
uuid: "779618f470734c48a21980d40f0bc144"
format_stamp: "Formatted at 2026-08-10 00:50:18 on dist-test-slave-1nl6"
server_key: "248aae53500d645c74e631738b0537f3"
server_key_iv: "52d616e3e8feee361d04041703ba6f4f"
server_key_version: "encryptionkey@0"
I20260810 00:50:18.293833 27134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance:
uuid: "779618f470734c48a21980d40f0bc144"
format_stamp: "Formatted at 2026-08-10 00:50:18 on dist-test-slave-1nl6"
server_key: "248aae53500d645c74e631738b0537f3"
server_key_iv: "52d616e3e8feee361d04041703ba6f4f"
server_key_version: "encryptionkey@0"
I20260810 00:50:18.298712 27134 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260810 00:50:18.301992 27154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:18.303260 27134 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 00:50:18.303437 27134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-2/wal
uuid: "779618f470734c48a21980d40f0bc144"
format_stamp: "Formatted at 2026-08-10 00:50:18 on dist-test-slave-1nl6"
server_key: "248aae53500d645c74e631738b0537f3"
server_key_iv: "52d616e3e8feee361d04041703ba6f4f"
server_key_version: "encryptionkey@0"
I20260810 00:50:18.303647 27134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:50:18.319315 27134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:50:18.320065 27134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:50:18.320348 27134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:50:18.321249 27134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 00:50:18.322677 27134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 00:50:18.322801 27134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:18.322958 27134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 00:50:18.323061 27134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:18.356794 27134 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.3:45307
I20260810 00:50:18.356848 27346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.3:45307 every 8 connection(s)
I20260810 00:50:18.358112 27134 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-2/data/info.pb
I20260810 00:50:18.366283 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 27134
I20260810 00:50:18.366461 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-2/wal/instance
I20260810 00:50:18.366940 28696 external_mini_cluster.cc:1428] Setting key 0ea084797a274e765ecc1b59a12f1dd9
I20260810 00:50:18.370842 26925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:18.360724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.3:49709 (local address 127.28.6.62:40399)
0810 00:50:18.360918 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:18.360923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:18.362207 (+ 1284us) server_negotiation.cc:407] Connection header received
0810 00:50:18.363988 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:18.363992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:18.364059 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:18.364144 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:18.365134 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:18.365870 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:18.367789 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:18.368412 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:18.368551 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:18.370153 (+ 1602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:18.370161 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:18.370166 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:18.370187 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:18.370245 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:18.370258 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:18.370328 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:18.370540 (+ 212us) server_negotiation.cc:299] Negotiation successful
0810 00:50:18.370611 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 00:50:18.371863 27350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:18.361208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:40399 (local address 127.28.6.3:49709)
0810 00:50:18.362066 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:18.362101 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:18.363743 (+ 1642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:18.364366 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:18.364372 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:18.364449 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:50:18.364969 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:18.364985 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:18.366000 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:18.366003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:18.367649 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:18.367663 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:18.368666 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:18.368678 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:18.368959 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:18.369687 (+ 728us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 00:50:18.369692 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:50:18.369747 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 00:50:18.369840 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:50:18.370356 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:50:18.370365 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:50:18.370370 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 00:50:18.370509 (+ 139us) client_negotiation.cc:240] Negotiation successful
0810 00:50:18.370770 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":259,"threads_started":1}
I20260810 00:50:18.373159 27347 heartbeater.cc:347] Connected to a master server at 127.28.6.62:40399
I20260810 00:50:18.373445 27347 heartbeater.cc:464] Registering TS with master...
I20260810 00:50:18.374178 27347 heartbeater.cc:511] Master 127.28.6.62:40399 requested a full tablet report, sending...
I20260810 00:50:18.375528 26630 ts_manager.cc:194] Registered new tserver with Master: 779618f470734c48a21980d40f0bc144 (127.28.6.3:45307)
I20260810 00:50:18.376349 26630 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.6.3:49709
I20260810 00:50:18.382084 28696 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 00:50:18.391649 26925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:18.386163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48182 (local address 127.28.6.62:40399)
0810 00:50:18.386396 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:18.386405 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:18.386734 (+ 329us) server_negotiation.cc:407] Connection header received
0810 00:50:18.386860 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:18.386864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:18.386997 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:18.387075 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:18.387540 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:18.388215 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:18.389579 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:18.389970 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:18.390062 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:18.391069 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:18.391073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:18.391077 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:18.391092 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:18.391119 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:18.391130 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:18.391195 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:18.391273 (+ 78us) server_negotiation.cc:299] Negotiation successful
0810 00:50:18.391344 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 00:50:18.395328 26630 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 00:50:18.402132 26630 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 00:50:18.436014 27368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:18.414039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:45307 (local address 127.0.0.1:54108)
0810 00:50:18.415485 (+ 1446us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:18.415522 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:18.415685 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:18.426922 (+ 11237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:18.426932 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:18.426964 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:50:18.427352 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:18.427367 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:18.429365 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:18.429370 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:18.431899 (+ 2529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:18.431921 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:18.434757 (+ 2836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:18.434766 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:18.435107 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:18.435184 (+ 77us) client_negotiation.cc:769] Sending connection context
0810 00:50:18.435321 (+ 137us) client_negotiation.cc:240] Negotiation successful
0810 00:50:18.435555 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1119,"thread_start_us":949,"threads_started":1}
I20260810 00:50:18.437085 27369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:18.414402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54108 (local address 127.28.6.3:45307)
0810 00:50:18.425929 (+ 11527us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:18.425941 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:18.425984 (+ 43us) server_negotiation.cc:407] Connection header received
0810 00:50:18.426091 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:18.426099 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:18.426337 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:18.426549 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:50:18.427595 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:18.429190 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:18.432207 (+ 3017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:18.434057 (+ 1850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:18.434256 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:18.434595 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:18.436536 (+ 1941us) server_negotiation.cc:299] Negotiation successful
0810 00:50:18.436747 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11291,"thread_start_us":11595,"threads_started":1}
I20260810 00:50:18.441193 27242 tablet_service.cc:1467] Processing CreateTablet for tablet bd26cf617550422892874898282533b4 (DEFAULT_TABLE table=test-table [id=5a3143990d524a5596662af3c8b9cac8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:50:18.442751 27242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd26cf617550422892874898282533b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:50:18.446012 27380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:18.415800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56150 (local address 127.28.6.1:38841)
0810 00:50:18.438696 (+ 22896us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:18.438703 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:18.438731 (+ 28us) server_negotiation.cc:407] Connection header received
0810 00:50:18.438816 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:18.438823 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:18.439084 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:18.439298 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:50:18.440270 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:18.441690 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:18.443939 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:18.444945 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:18.445098 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:18.445374 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:18.445486 (+ 112us) server_negotiation.cc:299] Negotiation successful
0810 00:50:18.445660 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22728,"thread_start_us":22603,"threads_started":1}
I20260810 00:50:18.446033 27376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:18.414948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.1:38841 (local address 127.0.0.1:56150)
0810 00:50:18.436969 (+ 22021us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:18.436987 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:18.437108 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:18.439550 (+ 2442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:18.439555 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:18.439585 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:50:18.440067 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:18.440082 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:18.441834 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:18.441839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:18.443693 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:18.443706 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:18.445133 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:18.445139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:18.445303 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:18.445343 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 00:50:18.445449 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 00:50:18.445594 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21901,"mutex_wait_us":66,"thread_start_us":22138,"threads_started":1}
I20260810 00:50:18.446586 27371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:18.414356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:44161 (local address 127.0.0.1:47690)
0810 00:50:18.428178 (+ 13822us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:18.428202 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:18.428350 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:18.439289 (+ 10939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:18.439294 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:18.439320 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:50:18.439713 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:18.439725 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:18.441505 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:18.441511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:18.443901 (+ 2390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:18.443917 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:18.445789 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:18.445796 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:18.446009 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:18.446052 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 00:50:18.446117 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 00:50:18.446238 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13664,"mutex_wait_us":88,"thread_start_us":16564,"threads_started":1}
I20260810 00:50:18.446858 26922 heartbeater.cc:503] Master 127.28.6.62:40399 was elected leader, sending a full tablet report...
I20260810 00:50:18.447831 27379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:18.414979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47690 (local address 127.28.6.2:44161)
0810 00:50:18.438644 (+ 23665us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:18.438652 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:18.438686 (+ 34us) server_negotiation.cc:407] Connection header received
0810 00:50:18.438769 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:18.438774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:18.439056 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:18.439304 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:50:18.439893 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:18.441355 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:18.444143 (+ 2788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:18.445547 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:18.447000 (+ 1453us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:18.447246 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:18.447313 (+ 67us) server_negotiation.cc:299] Negotiation successful
0810 00:50:18.447481 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23468,"thread_start_us":23975,"threads_started":1}
I20260810 00:50:18.449975 26811 tablet_service.cc:1467] Processing CreateTablet for tablet bd26cf617550422892874898282533b4 (DEFAULT_TABLE table=test-table [id=5a3143990d524a5596662af3c8b9cac8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:50:18.451449 26811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd26cf617550422892874898282533b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:50:18.451323 27022 tablet_service.cc:1467] Processing CreateTablet for tablet bd26cf617550422892874898282533b4 (DEFAULT_TABLE table=test-table [id=5a3143990d524a5596662af3c8b9cac8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 00:50:18.452699 27022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd26cf617550422892874898282533b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:50:18.456504 27382 tablet_bootstrap.cc:492] T bd26cf617550422892874898282533b4 P 779618f470734c48a21980d40f0bc144: Bootstrap starting.
I20260810 00:50:18.459820 27382 tablet_bootstrap.cc:654] T bd26cf617550422892874898282533b4 P 779618f470734c48a21980d40f0bc144: Neither blocks nor log segments found. Creating new log.
I20260810 00:50:18.460871 27382 log.cc:824] T bd26cf617550422892874898282533b4 P 779618f470734c48a21980d40f0bc144: Log is configured to *not* fsync() on all Append() calls
I20260810 00:50:18.463867 27382 tablet_bootstrap.cc:492] T bd26cf617550422892874898282533b4 P 779618f470734c48a21980d40f0bc144: No bootstrap required, opened a new log
I20260810 00:50:18.464145 27382 ts_tablet_manager.cc:1397] T bd26cf617550422892874898282533b4 P 779618f470734c48a21980d40f0bc144: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260810 00:50:18.466681 27387 tablet_bootstrap.cc:492] T bd26cf617550422892874898282533b4 P 25eda644720d4520a282b044bc54261a: Bootstrap starting.
I20260810 00:50:18.467531 27388 tablet_bootstrap.cc:492] T bd26cf617550422892874898282533b4 P 5b145e188b6647d3bdfa0484e4e317d2: Bootstrap starting.
I20260810 00:50:18.471143 27387 tablet_bootstrap.cc:654] T bd26cf617550422892874898282533b4 P 25eda644720d4520a282b044bc54261a: Neither blocks nor log segments found. Creating new log.
I20260810 00:50:18.472468 27387 log.cc:824] T bd26cf617550422892874898282533b4 P 25eda644720d4520a282b044bc54261a: Log is configured to *not* fsync() on all Append() calls
I20260810 00:50:18.473896 27388 tablet_bootstrap.cc:654] T bd26cf617550422892874898282533b4 P 5b145e188b6647d3bdfa0484e4e317d2: Neither blocks nor log segments found. Creating new log.
I20260810 00:50:18.474232 27382 raft_consensus.cc:348] T bd26cf617550422892874898282533b4 P 779618f470734c48a21980d40f0bc144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779618f470734c48a21980d40f0bc144" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45307 } } peers { permanent_uuid: "25eda644720d4520a282b044bc54261a" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 44161 } } peers { permanent_uuid: "5b145e188b6647d3bdfa0484e4e317d2" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 38841 } }
I20260810 00:50:18.474673 27382 raft_consensus.cc:374] T bd26cf617550422892874898282533b4 P 779618f470734c48a21980d40f0bc144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:50:18.474814 27382 raft_consensus.cc:729] T bd26cf617550422892874898282533b4 P 779618f470734c48a21980d40f0bc144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 779618f470734c48a21980d40f0bc144, State: Initialized, Role: FOLLOWER
I20260810 00:50:18.474951 27388 log.cc:824] T bd26cf617550422892874898282533b4 P 5b145e188b6647d3bdfa0484e4e317d2: Log is configured to *not* fsync() on all Append() calls
I20260810 00:50:18.475648 27387 tablet_bootstrap.cc:492] T bd26cf617550422892874898282533b4 P 25eda644720d4520a282b044bc54261a: No bootstrap required, opened a new log
I20260810 00:50:18.475593 27382 consensus_queue.cc:260] T bd26cf617550422892874898282533b4 P 779618f470734c48a21980d40f0bc144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779618f470734c48a21980d40f0bc144" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45307 } } peers { permanent_uuid: "25eda644720d4520a282b044bc54261a" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 44161 } } peers { permanent_uuid: "5b145e188b6647d3bdfa0484e4e317d2" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 38841 } }
I20260810 00:50:18.475919 27387 ts_tablet_manager.cc:1397] T bd26cf617550422892874898282533b4 P 25eda644720d4520a282b044bc54261a: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260810 00:50:18.478509 27382 ts_tablet_manager.cc:1428] T bd26cf617550422892874898282533b4 P 779618f470734c48a21980d40f0bc144: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
I20260810 00:50:18.478993 27388 tablet_bootstrap.cc:492] T bd26cf617550422892874898282533b4 P 5b145e188b6647d3bdfa0484e4e317d2: No bootstrap required, opened a new log
I20260810 00:50:18.479187 27347 heartbeater.cc:503] Master 127.28.6.62:40399 was elected leader, sending a full tablet report...
I20260810 00:50:18.479223 27388 ts_tablet_manager.cc:1397] T bd26cf617550422892874898282533b4 P 5b145e188b6647d3bdfa0484e4e317d2: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260810 00:50:18.484776 27387 raft_consensus.cc:348] T bd26cf617550422892874898282533b4 P 25eda644720d4520a282b044bc54261a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779618f470734c48a21980d40f0bc144" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45307 } } peers { permanent_uuid: "25eda644720d4520a282b044bc54261a" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 44161 } } peers { permanent_uuid: "5b145e188b6647d3bdfa0484e4e317d2" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 38841 } }
I20260810 00:50:18.485214 27387 raft_consensus.cc:374] T bd26cf617550422892874898282533b4 P 25eda644720d4520a282b044bc54261a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:50:18.485308 27387 raft_consensus.cc:729] T bd26cf617550422892874898282533b4 P 25eda644720d4520a282b044bc54261a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25eda644720d4520a282b044bc54261a, State: Initialized, Role: FOLLOWER
I20260810 00:50:18.485855 27387 consensus_queue.cc:260] T bd26cf617550422892874898282533b4 P 25eda644720d4520a282b044bc54261a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779618f470734c48a21980d40f0bc144" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45307 } } peers { permanent_uuid: "25eda644720d4520a282b044bc54261a" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 44161 } } peers { permanent_uuid: "5b145e188b6647d3bdfa0484e4e317d2" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 38841 } }
I20260810 00:50:18.486963 27130 heartbeater.cc:503] Master 127.28.6.62:40399 was elected leader, sending a full tablet report...
I20260810 00:50:18.487459 27387 ts_tablet_manager.cc:1428] T bd26cf617550422892874898282533b4 P 25eda644720d4520a282b044bc54261a: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260810 00:50:18.488126 27388 raft_consensus.cc:348] T bd26cf617550422892874898282533b4 P 5b145e188b6647d3bdfa0484e4e317d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779618f470734c48a21980d40f0bc144" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45307 } } peers { permanent_uuid: "25eda644720d4520a282b044bc54261a" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 44161 } } peers { permanent_uuid: "5b145e188b6647d3bdfa0484e4e317d2" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 38841 } }
I20260810 00:50:18.488493 27388 raft_consensus.cc:374] T bd26cf617550422892874898282533b4 P 5b145e188b6647d3bdfa0484e4e317d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:50:18.488600 27388 raft_consensus.cc:729] T bd26cf617550422892874898282533b4 P 5b145e188b6647d3bdfa0484e4e317d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b145e188b6647d3bdfa0484e4e317d2, State: Initialized, Role: FOLLOWER
I20260810 00:50:18.489176 27388 consensus_queue.cc:260] T bd26cf617550422892874898282533b4 P 5b145e188b6647d3bdfa0484e4e317d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779618f470734c48a21980d40f0bc144" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45307 } } peers { permanent_uuid: "25eda644720d4520a282b044bc54261a" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 44161 } } peers { permanent_uuid: "5b145e188b6647d3bdfa0484e4e317d2" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 38841 } }
I20260810 00:50:18.490805 27388 ts_tablet_manager.cc:1428] T bd26cf617550422892874898282533b4 P 5b145e188b6647d3bdfa0484e4e317d2: Time spent starting tablet: real 0.011s user 0.006s sys 0.005s
I20260810 00:50:18.592664 27398 raft_consensus.cc:482] T bd26cf617550422892874898282533b4 P 5b145e188b6647d3bdfa0484e4e317d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 00:50:18.593026 27398 raft_consensus.cc:504] T bd26cf617550422892874898282533b4 P 5b145e188b6647d3bdfa0484e4e317d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779618f470734c48a21980d40f0bc144" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45307 } } peers { permanent_uuid: "25eda644720d4520a282b044bc54261a" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 44161 } } peers { permanent_uuid: "5b145e188b6647d3bdfa0484e4e317d2" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 38841 } }
I20260810 00:50:18.594755 27398 leader_election.cc:288] T bd26cf617550422892874898282533b4 P 5b145e188b6647d3bdfa0484e4e317d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 779618f470734c48a21980d40f0bc144 (127.28.6.3:45307), 25eda644720d4520a282b044bc54261a (127.28.6.2:44161)
I20260810 00:50:18.601893 27379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:18.595258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:44597 (local address 127.28.6.2:44161)
0810 00:50:18.595486 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:18.595494 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:18.596375 (+ 881us) server_negotiation.cc:407] Connection header received
0810 00:50:18.596509 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:18.596514 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:18.596608 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:18.596704 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:50:18.597403 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:18.598132 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:18.600107 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:18.600897 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:18.600997 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:18.601091 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:18.601487 (+ 396us) server_negotiation.cc:299] Negotiation successful
0810 00:50:18.601646 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 00:50:18.601949 27399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:18.595331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.2:44161 (local address 127.28.6.1:44597)
0810 00:50:18.596252 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:18.596267 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:18.596386 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:18.596739 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:18.596743 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:18.596767 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:50:18.597228 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:18.597241 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:18.598272 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:18.598277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:18.599905 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:18.599918 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:18.601092 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:18.601100 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:18.601276 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:18.601345 (+ 69us) client_negotiation.cc:769] Sending connection context
0810 00:50:18.601416 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 00:50:18.601548 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"mutex_wait_us":20,"thread_start_us":494,"threads_started":1}
I20260810 00:50:18.602213 27369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:18.595462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.6.1:42863 (local address 127.28.6.3:45307)
0810 00:50:18.595727 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:18.595737 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:18.596527 (+ 790us) server_negotiation.cc:407] Connection header received
0810 00:50:18.596706 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:18.596712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:18.596785 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:18.596892 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 00:50:18.597531 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:18.598276 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:18.600221 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:18.601008 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:18.601136 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:18.601243 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:18.601618 (+ 375us) server_negotiation.cc:299] Negotiation successful
0810 00:50:18.601762 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260810 00:50:18.602955 27400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:18.595372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.3:45307 (local address 127.28.6.1:42863)
0810 00:50:18.596399 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0810 00:50:18.596417 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 00:50:18.596554 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:50:18.596921 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:50:18.596924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:50:18.596968 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 00:50:18.597336 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:18.597347 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:18.598420 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:18.598424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:18.600000 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:50:18.600040 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:50:18.601208 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:50:18.601213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:50:18.601379 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:18.601443 (+ 64us) client_negotiation.cc:769] Sending connection context
0810 00:50:18.602454 (+ 1011us) client_negotiation.cc:240] Negotiation successful
0810 00:50:18.602590 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":891,"mutex_wait_us":41,"spinlock_wait_cycles":2432,"thread_start_us":608,"threads_started":1}
I20260810 00:50:18.603137 27055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd26cf617550422892874898282533b4" candidate_uuid: "5b145e188b6647d3bdfa0484e4e317d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25eda644720d4520a282b044bc54261a" is_pre_election: true
I20260810 00:50:18.603746 27055 raft_consensus.cc:2454] T bd26cf617550422892874898282533b4 P 25eda644720d4520a282b044bc54261a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b145e188b6647d3bdfa0484e4e317d2 in term 0.
I20260810 00:50:18.603998 27274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd26cf617550422892874898282533b4" candidate_uuid: "5b145e188b6647d3bdfa0484e4e317d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "779618f470734c48a21980d40f0bc144" is_pre_election: true
I20260810 00:50:18.604511 26747 leader_election.cc:302] T bd26cf617550422892874898282533b4 P 5b145e188b6647d3bdfa0484e4e317d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25eda644720d4520a282b044bc54261a, 5b145e188b6647d3bdfa0484e4e317d2; no voters:
I20260810 00:50:18.604616 27274 raft_consensus.cc:2454] T bd26cf617550422892874898282533b4 P 779618f470734c48a21980d40f0bc144 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b145e188b6647d3bdfa0484e4e317d2 in term 0.
I20260810 00:50:18.604952 27398 raft_consensus.cc:2781] T bd26cf617550422892874898282533b4 P 5b145e188b6647d3bdfa0484e4e317d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 00:50:18.605144 27398 raft_consensus.cc:482] T bd26cf617550422892874898282533b4 P 5b145e188b6647d3bdfa0484e4e317d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 00:50:18.605260 27398 raft_consensus.cc:3037] T bd26cf617550422892874898282533b4 P 5b145e188b6647d3bdfa0484e4e317d2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:50:18.607764 27398 raft_consensus.cc:504] T bd26cf617550422892874898282533b4 P 5b145e188b6647d3bdfa0484e4e317d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779618f470734c48a21980d40f0bc144" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45307 } } peers { permanent_uuid: "25eda644720d4520a282b044bc54261a" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 44161 } } peers { permanent_uuid: "5b145e188b6647d3bdfa0484e4e317d2" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 38841 } }
I20260810 00:50:18.608397 27398 leader_election.cc:288] T bd26cf617550422892874898282533b4 P 5b145e188b6647d3bdfa0484e4e317d2 [CANDIDATE]: Term 1 election: Requested vote from peers 779618f470734c48a21980d40f0bc144 (127.28.6.3:45307), 25eda644720d4520a282b044bc54261a (127.28.6.2:44161)
I20260810 00:50:18.608789 27274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd26cf617550422892874898282533b4" candidate_uuid: "5b145e188b6647d3bdfa0484e4e317d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "779618f470734c48a21980d40f0bc144"
I20260810 00:50:18.608848 27055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd26cf617550422892874898282533b4" candidate_uuid: "5b145e188b6647d3bdfa0484e4e317d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25eda644720d4520a282b044bc54261a"
I20260810 00:50:18.608997 27274 raft_consensus.cc:3037] T bd26cf617550422892874898282533b4 P 779618f470734c48a21980d40f0bc144 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:50:18.609076 27055 raft_consensus.cc:3037] T bd26cf617550422892874898282533b4 P 25eda644720d4520a282b044bc54261a [term 0 FOLLOWER]: Advancing to term 1
W20260810 00:50:18.609848 27348 tablet.cc:2369] T bd26cf617550422892874898282533b4 P 779618f470734c48a21980d40f0bc144: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:50:18.611783 27055 raft_consensus.cc:2454] T bd26cf617550422892874898282533b4 P 25eda644720d4520a282b044bc54261a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b145e188b6647d3bdfa0484e4e317d2 in term 1.
I20260810 00:50:18.611950 27274 raft_consensus.cc:2454] T bd26cf617550422892874898282533b4 P 779618f470734c48a21980d40f0bc144 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b145e188b6647d3bdfa0484e4e317d2 in term 1.
I20260810 00:50:18.612185 26747 leader_election.cc:302] T bd26cf617550422892874898282533b4 P 5b145e188b6647d3bdfa0484e4e317d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25eda644720d4520a282b044bc54261a, 5b145e188b6647d3bdfa0484e4e317d2; no voters:
I20260810 00:50:18.612502 27398 raft_consensus.cc:2781] T bd26cf617550422892874898282533b4 P 5b145e188b6647d3bdfa0484e4e317d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:50:18.612950 27398 raft_consensus.cc:686] T bd26cf617550422892874898282533b4 P 5b145e188b6647d3bdfa0484e4e317d2 [term 1 LEADER]: Becoming Leader. State: Replica: 5b145e188b6647d3bdfa0484e4e317d2, State: Running, Role: LEADER
I20260810 00:50:18.613399 27398 consensus_queue.cc:237] T bd26cf617550422892874898282533b4 P 5b145e188b6647d3bdfa0484e4e317d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779618f470734c48a21980d40f0bc144" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45307 } } peers { permanent_uuid: "25eda644720d4520a282b044bc54261a" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 44161 } } peers { permanent_uuid: "5b145e188b6647d3bdfa0484e4e317d2" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 38841 } }
I20260810 00:50:18.618062 26629 catalog_manager.cc:5495] T bd26cf617550422892874898282533b4 P 5b145e188b6647d3bdfa0484e4e317d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b145e188b6647d3bdfa0484e4e317d2 (127.28.6.1). New cstate: current_term: 1 leader_uuid: "5b145e188b6647d3bdfa0484e4e317d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "779618f470734c48a21980d40f0bc144" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25eda644720d4520a282b044bc54261a" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 44161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b145e188b6647d3bdfa0484e4e317d2" member_type: VOTER last_known_addr { host: "127.28.6.1" port: 38841 } health_report { overall_health: HEALTHY } } }
W20260810 00:50:18.644047 27131 tablet.cc:2369] T bd26cf617550422892874898282533b4 P 25eda644720d4520a282b044bc54261a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:50:18.659763 27380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:18.654449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56152 (local address 127.28.6.1:38841)
0810 00:50:18.654688 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:18.654694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:18.654712 (+ 18us) server_negotiation.cc:407] Connection header received
0810 00:50:18.654931 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:18.654936 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:18.655034 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:18.655182 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:50:18.655592 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:18.656266 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:18.657630 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:18.658067 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:18.658195 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:18.658650 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:50:18.659199 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:50:18.659350 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:18.659454 (+ 104us) server_negotiation.cc:299] Negotiation successful
0810 00:50:18.659526 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 00:50:18.668008 27055 raft_consensus.cc:1261] T bd26cf617550422892874898282533b4 P 25eda644720d4520a282b044bc54261a [term 1 FOLLOWER]: Refusing update from remote peer 5b145e188b6647d3bdfa0484e4e317d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:50:18.668009 27274 raft_consensus.cc:1261] T bd26cf617550422892874898282533b4 P 779618f470734c48a21980d40f0bc144 [term 1 FOLLOWER]: Refusing update from remote peer 5b145e188b6647d3bdfa0484e4e317d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:50:18.669024 27398 consensus_queue.cc:1035] T bd26cf617550422892874898282533b4 P 5b145e188b6647d3bdfa0484e4e317d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "779618f470734c48a21980d40f0bc144" member_type: VOTER last_known_addr { host: "127.28.6.3" port: 45307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:50:18.669360 27402 consensus_queue.cc:1035] T bd26cf617550422892874898282533b4 P 5b145e188b6647d3bdfa0484e4e317d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25eda644720d4520a282b044bc54261a" member_type: VOTER last_known_addr { host: "127.28.6.2" port: 44161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 00:50:18.672293 26923 tablet.cc:2369] T bd26cf617550422892874898282533b4 P 5b145e188b6647d3bdfa0484e4e317d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:50:18.679188 27406 mvcc.cc:205] Tried to move back new op lower bound from 7316779084453441536 to 7316779084243935232. Current Snapshot: MvccSnapshot[applied={T|T < 7316779084453441536}]
I20260810 00:50:18.683014 27408 mvcc.cc:205] Tried to move back new op lower bound from 7316779084453441536 to 7316779084243935232. Current Snapshot: MvccSnapshot[applied={T|T < 7316779084453441536}]
I20260810 00:50:18.692550 27415 mvcc.cc:205] Tried to move back new op lower bound from 7316779084453441536 to 7316779084243935232. Current Snapshot: MvccSnapshot[applied={T|T < 7316779084453441536}]
I20260810 00:50:18.694712 26627 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:50:18.695053 26627 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 00:50:18.699134 26627 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c113b05ce77e498ebd5d30f8c17fcb93: Sending DeleteTablet for 3 replicas of tablet bd26cf617550422892874898282533b4
I20260810 00:50:18.700330 27022 tablet_service.cc:1514] Processing DeleteTablet for tablet bd26cf617550422892874898282533b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:50:18 UTC) from {username='slave'} at 127.0.0.1:47690
I20260810 00:50:18.700837 26811 tablet_service.cc:1514] Processing DeleteTablet for tablet bd26cf617550422892874898282533b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:50:18 UTC) from {username='slave'} at 127.0.0.1:56150
I20260810 00:50:18.701017 27242 tablet_service.cc:1514] Processing DeleteTablet for tablet bd26cf617550422892874898282533b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 00:50:18 UTC) from {username='slave'} at 127.0.0.1:54108
I20260810 00:50:18.701700 27425 tablet_replica.cc:330] T bd26cf617550422892874898282533b4 P 5b145e188b6647d3bdfa0484e4e317d2: stopping tablet replica
I20260810 00:50:18.702090 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 26725
I20260810 00:50:18.702153 27425 raft_consensus.cc:2229] T bd26cf617550422892874898282533b4 P 5b145e188b6647d3bdfa0484e4e317d2 [term 1 LEADER]: Raft consensus shutting down.
I20260810 00:50:18.702934 27425 raft_consensus.cc:2258] T bd26cf617550422892874898282533b4 P 5b145e188b6647d3bdfa0484e4e317d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:50:18.705075 27425 ts_tablet_manager.cc:1905] T bd26cf617550422892874898282533b4 P 5b145e188b6647d3bdfa0484e4e317d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:50:18.708191 27423 tablet_replica.cc:330] T bd26cf617550422892874898282533b4 P 25eda644720d4520a282b044bc54261a: stopping tablet replica
I20260810 00:50:18.708622 27423 raft_consensus.cc:2229] T bd26cf617550422892874898282533b4 P 25eda644720d4520a282b044bc54261a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:50:18.709091 27423 raft_consensus.cc:2258] T bd26cf617550422892874898282533b4 P 25eda644720d4520a282b044bc54261a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:50:18.709343 27427 tablet_replica.cc:330] T bd26cf617550422892874898282533b4 P 779618f470734c48a21980d40f0bc144: stopping tablet replica
I20260810 00:50:18.709781 27427 raft_consensus.cc:2229] T bd26cf617550422892874898282533b4 P 779618f470734c48a21980d40f0bc144 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 00:50:18.710287 27427 raft_consensus.cc:2258] T bd26cf617550422892874898282533b4 P 779618f470734c48a21980d40f0bc144 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 00:50:18.712772 27427 ts_tablet_manager.cc:1905] T bd26cf617550422892874898282533b4 P 779618f470734c48a21980d40f0bc144: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:50:18.713176 27423 ts_tablet_manager.cc:1905] T bd26cf617550422892874898282533b4 P 25eda644720d4520a282b044bc54261a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 00:50:18.713428 27425 ts_tablet_manager.cc:1918] T bd26cf617550422892874898282533b4 P 5b145e188b6647d3bdfa0484e4e317d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:50:18.713589 27425 log.cc:1196] T bd26cf617550422892874898282533b4 P 5b145e188b6647d3bdfa0484e4e317d2: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-0/wal/wals/bd26cf617550422892874898282533b4
I20260810 00:50:18.714178 27425 ts_tablet_manager.cc:1939] T bd26cf617550422892874898282533b4 P 5b145e188b6647d3bdfa0484e4e317d2: Deleting consensus metadata
I20260810 00:50:18.716363 26602 catalog_manager.cc:4841] TS 5b145e188b6647d3bdfa0484e4e317d2 (127.28.6.1:38841): tablet bd26cf617550422892874898282533b4 (table test-table [id=5a3143990d524a5596662af3c8b9cac8]) successfully deleted
I20260810 00:50:18.717888 27380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:18.704943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56160 (local address 127.28.6.1:38841)
0810 00:50:18.705426 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:18.705434 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:18.709041 (+ 3607us) server_negotiation.cc:407] Connection header received
0810 00:50:18.709305 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:18.709312 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:18.709485 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:18.709588 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:18.710420 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:18.711733 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:18.713702 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:18.714407 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:18.714505 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:18.716014 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:18.716025 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:18.716042 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:18.716865 (+ 823us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:18.716924 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:18.716949 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:18.717049 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:18.717397 (+ 348us) server_negotiation.cc:299] Negotiation successful
0810 00:50:18.717509 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323}
I20260810 00:50:18.719074 26913 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:50:18.720312 27427 ts_tablet_manager.cc:1918] T bd26cf617550422892874898282533b4 P 779618f470734c48a21980d40f0bc144: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:50:18.720494 27427 log.cc:1196] T bd26cf617550422892874898282533b4 P 779618f470734c48a21980d40f0bc144: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-2/wal/wals/bd26cf617550422892874898282533b4
I20260810 00:50:18.721081 27427 ts_tablet_manager.cc:1939] T bd26cf617550422892874898282533b4 P 779618f470734c48a21980d40f0bc144: Deleting consensus metadata
I20260810 00:50:18.723088 26607 catalog_manager.cc:4841] TS 779618f470734c48a21980d40f0bc144 (127.28.6.3:45307): tablet bd26cf617550422892874898282533b4 (table test-table [id=5a3143990d524a5596662af3c8b9cac8]) successfully deleted
I20260810 00:50:18.723948 27423 ts_tablet_manager.cc:1918] T bd26cf617550422892874898282533b4 P 25eda644720d4520a282b044bc54261a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 00:50:18.724102 27423 log.cc:1196] T bd26cf617550422892874898282533b4 P 25eda644720d4520a282b044bc54261a: Deleting WAL directory at /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786322687226232-28696-0/minicluster-data/ts-1/wal/wals/bd26cf617550422892874898282533b4
I20260810 00:50:18.724558 27423 ts_tablet_manager.cc:1939] T bd26cf617550422892874898282533b4 P 25eda644720d4520a282b044bc54261a: Deleting consensus metadata
I20260810 00:50:18.725950 26606 catalog_manager.cc:4841] TS 25eda644720d4520a282b044bc54261a (127.28.6.2:44161): tablet bd26cf617550422892874898282533b4 (table test-table [id=5a3143990d524a5596662af3c8b9cac8]) successfully deleted
I20260810 00:50:18.812376 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 26725
I20260810 00:50:18.841552 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 26929
I20260810 00:50:18.848706 27379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:18.842360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47694 (local address 127.28.6.2:44161)
0810 00:50:18.842538 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:18.842542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:18.842560 (+ 18us) server_negotiation.cc:407] Connection header received
0810 00:50:18.842700 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:18.842704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:18.842793 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:18.842889 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:18.843326 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:18.844109 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:18.845340 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:18.845709 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:18.845796 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:18.846763 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:18.846771 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:18.846787 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:18.847775 (+ 988us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:18.847847 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:18.847886 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:18.848030 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:18.848212 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 00:50:18.848335 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 00:50:18.849423 27123 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:50:18.938459 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 26929
I20260810 00:50:18.961529 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 27134
I20260810 00:50:18.968164 27369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:18.962300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54124 (local address 127.28.6.3:45307)
0810 00:50:18.962473 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:18.962477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:18.962528 (+ 51us) server_negotiation.cc:407] Connection header received
0810 00:50:18.962685 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:18.962689 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:18.962797 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:18.962882 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:18.963333 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:18.964060 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:18.965303 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:18.965673 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:18.965775 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:18.966700 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:18.966709 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:18.966734 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:18.967522 (+ 788us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:18.967573 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:18.967593 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:18.967683 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:18.967819 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 00:50:18.967915 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 00:50:18.968818 27341 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:50:19.055884 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 27134
I20260810 00:50:19.081238 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 26580
I20260810 00:50:19.082391 26679 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:50:19.189841 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 26580
2026-08-10T00:50:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2843 ms)
[----------] 6 tests from EncryptionPolicyTest (22948 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:50:19 dist-test-slave-1nl6 krb5kdc[27437](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:50:19 dist-test-slave-1nl6 krb5kdc[27437](info): set up 2 sockets
Aug 10 00:50:19 dist-test-slave-1nl6 krb5kdc[27437](info): commencing operation
krb5kdc: starting...
W20260810 00:50:20.520915 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.277s user 0.003s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:50:20 dist-test-slave-1nl6 krb5kdc[27437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:50:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:50:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 00:50:20.808768 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43973
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:50:21.143947 27452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:50:21.144246 27452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:50:21.144423 27452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:50:21.151580 27452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:50:21.151717 27452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:50:21.151808 27452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:50:21.151909 27452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:50:21.151996 27452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:50:21.161646 27452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43973
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:50:21.162941 27452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:50:21.164309 27452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:50:21.176533 27461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:21.176581 27468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:21.176829 27462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:50:21.177513 27452 server_base.cc:1034] running on GCE node
I20260810 00:50:21.178217 27452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:50:21.179636 27452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:50:21.180876 27452 hybrid_clock.cc:648] HybridClock initialized: now 1786323021180832 us; error 51 us; skew 500 ppm
I20260810 00:50:21.181224 27452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:50:21 dist-test-slave-1nl6 krb5kdc[27437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323021, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:50:21.184408 27452 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 00:50:21.185750 27452 webserver.cc:460] Webserver started at http://127.0.0.1:35291/ using document root <none> and password file <none>
I20260810 00:50:21.186367 27452 fs_manager.cc:359] Metadata directory not provided
I20260810 00:50:21.186473 27452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:50:21.186723 27452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:50:21.189668 27452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "8a8e626af01349f281ae77234a095d8f"
format_stamp: "Formatted at 2026-08-10 00:50:21 on dist-test-slave-1nl6"
server_key: "3fe8d8babf3c32dbd49cc9b7e3dc7776"
server_key_iv: "b443307f94ef1472921b91ca1f09b85c"
server_key_version: "encryptionkey@0"
I20260810 00:50:21.190407 27452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "8a8e626af01349f281ae77234a095d8f"
format_stamp: "Formatted at 2026-08-10 00:50:21 on dist-test-slave-1nl6"
server_key: "3fe8d8babf3c32dbd49cc9b7e3dc7776"
server_key_iv: "b443307f94ef1472921b91ca1f09b85c"
server_key_version: "encryptionkey@0"
I20260810 00:50:21.195436 27452 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260810 00:50:21.198762 27481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:21.200094 27452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 00:50:21.200234 27452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "8a8e626af01349f281ae77234a095d8f"
format_stamp: "Formatted at 2026-08-10 00:50:21 on dist-test-slave-1nl6"
server_key: "3fe8d8babf3c32dbd49cc9b7e3dc7776"
server_key_iv: "b443307f94ef1472921b91ca1f09b85c"
server_key_version: "encryptionkey@0"
I20260810 00:50:21.200403 27452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:50:21.224274 27452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:50:21.228682 27452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:50:21.228930 27452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:50:21.247646 27452 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43973
I20260810 00:50:21.247727 27557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43973 every 8 connection(s)
I20260810 00:50:21.248988 27452 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:50:21.252635 27560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:50:21.255479 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 27452
I20260810 00:50:21.255728 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:50:21.256477 28696 external_mini_cluster.cc:1428] Setting key 15c2f290951618f1feb6e39dc9f65d5c
I20260810 00:50:21.265092 27560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a8e626af01349f281ae77234a095d8f: Bootstrap starting.
I20260810 00:50:21.268903 27560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a8e626af01349f281ae77234a095d8f: Neither blocks nor log segments found. Creating new log.
I20260810 00:50:21.270238 27560 log.cc:824] T 00000000000000000000000000000000 P 8a8e626af01349f281ae77234a095d8f: Log is configured to *not* fsync() on all Append() calls
Aug 10 00:50:21 dist-test-slave-1nl6 krb5kdc[27437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 00:50:21.273422 27560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a8e626af01349f281ae77234a095d8f: No bootstrap required, opened a new log
I20260810 00:50:21.280310 27569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:21.258513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36439 (local address 127.0.0.1:43973)
0810 00:50:21.259348 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:21.259359 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:21.259401 (+ 42us) server_negotiation.cc:407] Connection header received
0810 00:50:21.261067 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:21.261088 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:21.261477 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:21.261788 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:21.263614 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:21.264867 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:21.266853 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:21.267552 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:21.267686 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:21.272646 (+ 4960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:21.272657 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:21.272675 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:50:21.272724 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:21.276140 (+ 3416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:21.276957 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:21.276964 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:21.276970 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:21.277138 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:21.277630 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:21.277634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:21.277636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:21.278138 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:21.278511 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:21.279041 (+ 530us) server_negotiation.cc:299] Negotiation successful
0810 00:50:21.279354 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":416,"threads_started":1}
I20260810 00:50:21.281471 27560 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a8e626af01349f281ae77234a095d8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8e626af01349f281ae77234a095d8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43973 } }
I20260810 00:50:21.281904 27560 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a8e626af01349f281ae77234a095d8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:50:21.282023 27560 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a8e626af01349f281ae77234a095d8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a8e626af01349f281ae77234a095d8f, State: Initialized, Role: FOLLOWER
I20260810 00:50:21.282572 27560 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a8e626af01349f281ae77234a095d8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8e626af01349f281ae77234a095d8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43973 } }
I20260810 00:50:21.282833 27560 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a8e626af01349f281ae77234a095d8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:50:21.283033 27560 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a8e626af01349f281ae77234a095d8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:50:21.283231 27560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a8e626af01349f281ae77234a095d8f [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:50:21.284754 28696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 00:50:21.285676 27560 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a8e626af01349f281ae77234a095d8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8e626af01349f281ae77234a095d8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43973 } }
I20260810 00:50:21.286147 27560 leader_election.cc:302] T 00000000000000000000000000000000 P 8a8e626af01349f281ae77234a095d8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a8e626af01349f281ae77234a095d8f; no voters:
I20260810 00:50:21.286769 27560 leader_election.cc:288] T 00000000000000000000000000000000 P 8a8e626af01349f281ae77234a095d8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:50:21.286953 27572 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a8e626af01349f281ae77234a095d8f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:50:21.287618 27572 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a8e626af01349f281ae77234a095d8f [term 1 LEADER]: Becoming Leader. State: Replica: 8a8e626af01349f281ae77234a095d8f, State: Running, Role: LEADER
I20260810 00:50:21.288347 27572 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a8e626af01349f281ae77234a095d8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8e626af01349f281ae77234a095d8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43973 } }
I20260810 00:50:21.290593 27572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a8e626af01349f281ae77234a095d8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a8e626af01349f281ae77234a095d8f. Latest consensus state: current_term: 1 leader_uuid: "8a8e626af01349f281ae77234a095d8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8e626af01349f281ae77234a095d8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43973 } } }
I20260810 00:50:21.290934 27572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a8e626af01349f281ae77234a095d8f [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:21.291589 27575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a8e626af01349f281ae77234a095d8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a8e626af01349f281ae77234a095d8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8e626af01349f281ae77234a095d8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43973 } } }
I20260810 00:50:21.291991 27575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a8e626af01349f281ae77234a095d8f [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:21.300169 27560 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a8e626af01349f281ae77234a095d8f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:50:21.300355 27581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:50:21.306819 27581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:50:21.319619 27569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:21.300895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37263 (local address 127.0.0.1:43973)
0810 00:50:21.301109 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:21.301117 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:21.301496 (+ 379us) server_negotiation.cc:407] Connection header received
0810 00:50:21.301800 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:21.301806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:21.301878 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:21.301935 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:21.304152 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:21.305460 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:21.307490 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:21.308287 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:21.308430 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:21.311868 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:21.311875 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:21.311880 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:50:21.311916 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:21.315754 (+ 3838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:21.316781 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:21.316789 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:21.316792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:21.316968 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:21.317600 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:21.317605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:21.317608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:21.318007 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:21.318302 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:21.319091 (+ 789us) server_negotiation.cc:299] Negotiation successful
0810 00:50:21.319278 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 00:50:21.320757 27581 catalog_manager.cc:1347] Generated new cluster ID: 02626ea9b1e44a089b5d028c67550c47
I20260810 00:50:21.320874 27581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:50:21.336037 27581 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:50:21.337713 27581 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:50:21.351022 27581 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a8e626af01349f281ae77234a095d8f: Generated new TSK 0
I20260810 00:50:21.351943 27581 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 00:50:21.359736 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 27452
I20260810 00:50:21.360960 27548 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:50:21.449352 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 27452
W20260810 00:50:21.472401 27435 connection.cc:535] client connection to 127.0.0.1:43973 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43973): Connection reset by peer (error 104)
2026-08-10T00:50:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2248 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:50:21 dist-test-slave-1nl6 krb5kdc[27615](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:50:21 dist-test-slave-1nl6 krb5kdc[27615](info): set up 2 sockets
Aug 10 00:50:21 dist-test-slave-1nl6 krb5kdc[27615](info): commencing operation
krb5kdc: starting...
W20260810 00:50:22.755937 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.264s user 0.001s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:50:22 dist-test-slave-1nl6 krb5kdc[27615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:50:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:50:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 00:50:23.048228 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43981
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:50:23.383879 27640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:50:23.384223 27640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:50:23.384390 27640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:50:23.391577 27640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:50:23.391728 27640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:50:23.391832 27640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:50:23.391947 27640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:50:23.392024 27640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:50:23.401597 27640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43981
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:50:23.402890 27640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:50:23.404241 27640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:50:23.416435 27649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:23.416436 27646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:23.416467 27647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:50:23.417603 27640 server_base.cc:1034] running on GCE node
I20260810 00:50:23.418326 27640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:50:23.419736 27640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:50:23.420977 27640 hybrid_clock.cc:648] HybridClock initialized: now 1786323023420937 us; error 50 us; skew 500 ppm
I20260810 00:50:23.421319 27640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:50:23 dist-test-slave-1nl6 krb5kdc[27615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:50:23.428638 27640 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 00:50:23.430125 27640 webserver.cc:460] Webserver started at http://127.0.0.1:37203/ using document root <none> and password file <none>
I20260810 00:50:23.430771 27640 fs_manager.cc:359] Metadata directory not provided
I20260810 00:50:23.430968 27640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:50:23.431277 27640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:50:23.434197 27640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "ad29fe91c18f4e1a9b089c82380d3ec3"
format_stamp: "Formatted at 2026-08-10 00:50:23 on dist-test-slave-1nl6"
server_key: "3827fc5b872575a582e6715c57ab228c"
server_key_iv: "5280658c7d7a140a57c13b37e2e3251c"
server_key_version: "encryptionkey@0"
I20260810 00:50:23.435043 27640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "ad29fe91c18f4e1a9b089c82380d3ec3"
format_stamp: "Formatted at 2026-08-10 00:50:23 on dist-test-slave-1nl6"
server_key: "3827fc5b872575a582e6715c57ab228c"
server_key_iv: "5280658c7d7a140a57c13b37e2e3251c"
server_key_version: "encryptionkey@0"
I20260810 00:50:23.439945 27640 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260810 00:50:23.443216 27657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:23.444473 27640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 00:50:23.444662 27640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "ad29fe91c18f4e1a9b089c82380d3ec3"
format_stamp: "Formatted at 2026-08-10 00:50:23 on dist-test-slave-1nl6"
server_key: "3827fc5b872575a582e6715c57ab228c"
server_key_iv: "5280658c7d7a140a57c13b37e2e3251c"
server_key_version: "encryptionkey@0"
I20260810 00:50:23.444864 27640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:50:23.471643 27640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:50:23.475510 27640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:50:23.475797 27640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:50:23.494298 27640 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43981
I20260810 00:50:23.494346 27745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43981 every 8 connection(s)
I20260810 00:50:23.495710 27640 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:50:23.499397 27747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:50:23.505807 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 27640
I20260810 00:50:23.505995 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:50:23.506453 28696 external_mini_cluster.cc:1428] Setting key 120dd671ad0f5f8fa8cc5b767d8108a6
I20260810 00:50:23.509723 27747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad29fe91c18f4e1a9b089c82380d3ec3: Bootstrap starting.
I20260810 00:50:23.513058 27747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad29fe91c18f4e1a9b089c82380d3ec3: Neither blocks nor log segments found. Creating new log.
I20260810 00:50:23.514196 27747 log.cc:824] T 00000000000000000000000000000000 P ad29fe91c18f4e1a9b089c82380d3ec3: Log is configured to *not* fsync() on all Append() calls
I20260810 00:50:23.517282 27747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad29fe91c18f4e1a9b089c82380d3ec3: No bootstrap required, opened a new log
I20260810 00:50:23.520035 27750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:23.508393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56833 (local address 127.0.0.1:43981)
0810 00:50:23.509176 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:23.509188 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:23.509230 (+ 42us) server_negotiation.cc:407] Connection header received
0810 00:50:23.511049 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:23.511076 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:23.511463 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:23.511767 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:23.512464 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:23.513816 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:23.515626 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:23.516277 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:23.516410 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:23.517729 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:23.517740 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:23.517758 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:23.518276 (+ 518us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:23.518357 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:23.518387 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:23.518509 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:23.518816 (+ 307us) server_negotiation.cc:299] Negotiation successful
0810 00:50:23.519061 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":374,"threads_started":1}
I20260810 00:50:23.523875 28696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 00:50:23.525383 27747 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad29fe91c18f4e1a9b089c82380d3ec3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad29fe91c18f4e1a9b089c82380d3ec3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43981 } }
I20260810 00:50:23.525719 27747 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad29fe91c18f4e1a9b089c82380d3ec3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:50:23.525820 27747 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad29fe91c18f4e1a9b089c82380d3ec3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad29fe91c18f4e1a9b089c82380d3ec3, State: Initialized, Role: FOLLOWER
I20260810 00:50:23.526438 27747 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad29fe91c18f4e1a9b089c82380d3ec3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad29fe91c18f4e1a9b089c82380d3ec3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43981 } }
I20260810 00:50:23.526747 27747 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad29fe91c18f4e1a9b089c82380d3ec3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:50:23.526933 27747 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad29fe91c18f4e1a9b089c82380d3ec3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:50:23.527134 27747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad29fe91c18f4e1a9b089c82380d3ec3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:50:23.529587 27747 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad29fe91c18f4e1a9b089c82380d3ec3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad29fe91c18f4e1a9b089c82380d3ec3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43981 } }
I20260810 00:50:23.530072 27747 leader_election.cc:302] T 00000000000000000000000000000000 P ad29fe91c18f4e1a9b089c82380d3ec3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad29fe91c18f4e1a9b089c82380d3ec3; no voters:
I20260810 00:50:23.530757 27747 leader_election.cc:288] T 00000000000000000000000000000000 P ad29fe91c18f4e1a9b089c82380d3ec3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:50:23.530877 27766 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad29fe91c18f4e1a9b089c82380d3ec3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:50:23.533249 27747 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad29fe91c18f4e1a9b089c82380d3ec3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:50:23.535861 27750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:23.528441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44905 (local address 127.0.0.1:43981)
0810 00:50:23.528682 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:23.528691 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:23.529233 (+ 542us) server_negotiation.cc:407] Connection header received
0810 00:50:23.529424 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:23.529429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:23.529494 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:23.529604 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:23.530181 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:23.531097 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:23.532853 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:23.533316 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:23.533464 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:23.534673 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:23.534679 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:23.534684 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:23.534710 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:23.534802 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:23.534834 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:23.535024 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:23.535528 (+ 504us) server_negotiation.cc:299] Negotiation successful
0810 00:50:23.535583 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 00:50:23.537878 27766 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad29fe91c18f4e1a9b089c82380d3ec3 [term 1 LEADER]: Becoming Leader. State: Replica: ad29fe91c18f4e1a9b089c82380d3ec3, State: Running, Role: LEADER
I20260810 00:50:23.538529 27766 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad29fe91c18f4e1a9b089c82380d3ec3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad29fe91c18f4e1a9b089c82380d3ec3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43981 } }
I20260810 00:50:23.541780 27767 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad29fe91c18f4e1a9b089c82380d3ec3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad29fe91c18f4e1a9b089c82380d3ec3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad29fe91c18f4e1a9b089c82380d3ec3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43981 } } }
I20260810 00:50:23.542189 27767 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad29fe91c18f4e1a9b089c82380d3ec3 [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:23.542645 27768 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad29fe91c18f4e1a9b089c82380d3ec3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad29fe91c18f4e1a9b089c82380d3ec3. Latest consensus state: current_term: 1 leader_uuid: "ad29fe91c18f4e1a9b089c82380d3ec3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad29fe91c18f4e1a9b089c82380d3ec3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43981 } } }
I20260810 00:50:23.543072 27768 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad29fe91c18f4e1a9b089c82380d3ec3 [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:23.544103 27782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:50:23.551923 27782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:50:23.561079 27782 catalog_manager.cc:1347] Generated new cluster ID: 59eb8007aa62405398096c09e843f295
I20260810 00:50:23.561197 27782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:50:23.573582 27782 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:50:23.574872 27782 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:50:23.581969 27782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad29fe91c18f4e1a9b089c82380d3ec3: Generated new TSK 0
I20260810 00:50:23.582697 27782 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 00:50:23.592520 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 27640
W20260810 00:50:23.593645 27740 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56833
W20260810 00:50:23.594540 28696 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 27640: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 00:50:23.594728 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 27640
W20260810 00:50:23.620292 27613 connection.cc:535] client connection to 127.0.0.1:43981 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43981): Connection reset by peer (error 104)
2026-08-10T00:50:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2147 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:50:23 dist-test-slave-1nl6 krb5kdc[27795](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:50:23 dist-test-slave-1nl6 krb5kdc[27795](info): set up 2 sockets
Aug 10 00:50:23 dist-test-slave-1nl6 krb5kdc[27795](info): commencing operation
krb5kdc: starting...
W20260810 00:50:24.913450 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.274s user 0.002s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:50:25 dist-test-slave-1nl6 krb5kdc[27795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:50:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:50:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 00:50:25.197817 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44353
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:50:25.528868 27824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:50:25.529120 27824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:50:25.529297 27824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:50:25.536403 27824 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:50:25.536532 27824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:50:25.536623 27824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:50:25.536726 27824 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:50:25.536803 27824 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:50:25.546208 27824 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44353
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:50:25.547488 27824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:50:25.548745 27824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:50:25.561589 27829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:25.561559 27830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:25.561959 27835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:50:25.562382 27824 server_base.cc:1034] running on GCE node
I20260810 00:50:25.563206 27824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:50:25.564530 27824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:50:25.565773 27824 hybrid_clock.cc:648] HybridClock initialized: now 1786323025565724 us; error 57 us; skew 500 ppm
I20260810 00:50:25.566146 27824 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:50:25 dist-test-slave-1nl6 krb5kdc[27795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:50:25.569373 27824 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 00:50:25.570739 27824 webserver.cc:460] Webserver started at http://127.0.0.1:43833/ using document root <none> and password file <none>
I20260810 00:50:25.571465 27824 fs_manager.cc:359] Metadata directory not provided
I20260810 00:50:25.571612 27824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:50:25.571938 27824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:50:25.574822 27824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "c2bdfd6f97ee4f02ab61cdcc6c32d44b"
format_stamp: "Formatted at 2026-08-10 00:50:25 on dist-test-slave-1nl6"
server_key: "78eec50ed37c068697a253b9fddd1961"
server_key_iv: "dbbbff7a0af615c2e9606509ae24568a"
server_key_version: "encryptionkey@0"
I20260810 00:50:25.575677 27824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "c2bdfd6f97ee4f02ab61cdcc6c32d44b"
format_stamp: "Formatted at 2026-08-10 00:50:25 on dist-test-slave-1nl6"
server_key: "78eec50ed37c068697a253b9fddd1961"
server_key_iv: "dbbbff7a0af615c2e9606509ae24568a"
server_key_version: "encryptionkey@0"
I20260810 00:50:25.580787 27824 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 00:50:25.584060 27846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:25.585256 27824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 00:50:25.585443 27824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "c2bdfd6f97ee4f02ab61cdcc6c32d44b"
format_stamp: "Formatted at 2026-08-10 00:50:25 on dist-test-slave-1nl6"
server_key: "78eec50ed37c068697a253b9fddd1961"
server_key_iv: "dbbbff7a0af615c2e9606509ae24568a"
server_key_version: "encryptionkey@0"
I20260810 00:50:25.585642 27824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:50:25.591245 27824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:50:25.595655 27824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:50:25.595966 27824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:50:25.613838 27824 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44353
I20260810 00:50:25.613922 27942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44353 every 8 connection(s)
I20260810 00:50:25.615216 27824 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:50:25.618680 27943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:50:25.621356 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 27824
I20260810 00:50:25.621549 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:50:25.622081 28696 external_mini_cluster.cc:1428] Setting key 52c4ef24f9562cacbd887993d7f7334b
I20260810 00:50:25.629580 27946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:25.623803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51747 (local address 127.0.0.1:44353)
0810 00:50:25.624663 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:25.624675 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:25.624715 (+ 40us) server_negotiation.cc:407] Connection header received
0810 00:50:25.626349 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:25.626370 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:25.626690 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:25.627091 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:25.628079 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:25.628085 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:25.628100 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:25.628114 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:25.628169 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:25.628191 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:25.628356 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:25.628471 (+ 115us) server_negotiation.cc:299] Negotiation successful
0810 00:50:25.628689 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":436,"threads_started":1}
I20260810 00:50:25.630643 27943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2bdfd6f97ee4f02ab61cdcc6c32d44b: Bootstrap starting.
I20260810 00:50:25.633163 28696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 00:50:25.633913 27943 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2bdfd6f97ee4f02ab61cdcc6c32d44b: Neither blocks nor log segments found. Creating new log.
I20260810 00:50:25.635131 27943 log.cc:824] T 00000000000000000000000000000000 P c2bdfd6f97ee4f02ab61cdcc6c32d44b: Log is configured to *not* fsync() on all Append() calls
I20260810 00:50:25.638345 27943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2bdfd6f97ee4f02ab61cdcc6c32d44b: No bootstrap required, opened a new log
I20260810 00:50:25.640730 27946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:25.638273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35695 (local address 127.0.0.1:44353)
0810 00:50:25.638455 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:25.638461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:25.638826 (+ 365us) server_negotiation.cc:407] Connection header received
0810 00:50:25.638999 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:25.639004 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:25.639063 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:25.639171 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:25.640032 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:25.640042 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:25.640047 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:25.640065 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:25.640110 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:25.640160 (+ 50us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:25.640320 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:25.640379 (+ 59us) server_negotiation.cc:299] Negotiation successful
0810 00:50:25.640431 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 00:50:25.645856 27943 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2bdfd6f97ee4f02ab61cdcc6c32d44b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2bdfd6f97ee4f02ab61cdcc6c32d44b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44353 } }
I20260810 00:50:25.646188 27943 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2bdfd6f97ee4f02ab61cdcc6c32d44b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:50:25.646284 27943 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2bdfd6f97ee4f02ab61cdcc6c32d44b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2bdfd6f97ee4f02ab61cdcc6c32d44b, State: Initialized, Role: FOLLOWER
I20260810 00:50:25.646829 27943 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2bdfd6f97ee4f02ab61cdcc6c32d44b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2bdfd6f97ee4f02ab61cdcc6c32d44b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44353 } }
I20260810 00:50:25.647110 27943 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2bdfd6f97ee4f02ab61cdcc6c32d44b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:50:25.647264 27943 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2bdfd6f97ee4f02ab61cdcc6c32d44b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:50:25.647431 27943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2bdfd6f97ee4f02ab61cdcc6c32d44b [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:50:25.649407 27943 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2bdfd6f97ee4f02ab61cdcc6c32d44b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2bdfd6f97ee4f02ab61cdcc6c32d44b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44353 } }
I20260810 00:50:25.649875 27943 leader_election.cc:302] T 00000000000000000000000000000000 P c2bdfd6f97ee4f02ab61cdcc6c32d44b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2bdfd6f97ee4f02ab61cdcc6c32d44b; no voters:
I20260810 00:50:25.650449 27943 leader_election.cc:288] T 00000000000000000000000000000000 P c2bdfd6f97ee4f02ab61cdcc6c32d44b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:50:25.650576 27959 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2bdfd6f97ee4f02ab61cdcc6c32d44b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:50:25.651342 27959 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2bdfd6f97ee4f02ab61cdcc6c32d44b [term 1 LEADER]: Becoming Leader. State: Replica: c2bdfd6f97ee4f02ab61cdcc6c32d44b, State: Running, Role: LEADER
I20260810 00:50:25.651857 27959 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2bdfd6f97ee4f02ab61cdcc6c32d44b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2bdfd6f97ee4f02ab61cdcc6c32d44b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44353 } }
I20260810 00:50:25.652501 27943 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2bdfd6f97ee4f02ab61cdcc6c32d44b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:50:25.655340 27960 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2bdfd6f97ee4f02ab61cdcc6c32d44b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2bdfd6f97ee4f02ab61cdcc6c32d44b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2bdfd6f97ee4f02ab61cdcc6c32d44b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44353 } } }
I20260810 00:50:25.655309 27961 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2bdfd6f97ee4f02ab61cdcc6c32d44b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2bdfd6f97ee4f02ab61cdcc6c32d44b. Latest consensus state: current_term: 1 leader_uuid: "c2bdfd6f97ee4f02ab61cdcc6c32d44b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2bdfd6f97ee4f02ab61cdcc6c32d44b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44353 } } }
I20260810 00:50:25.655846 27960 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2bdfd6f97ee4f02ab61cdcc6c32d44b [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:25.655867 27961 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2bdfd6f97ee4f02ab61cdcc6c32d44b [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:25.659502 27966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:50:25.664326 27966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:50:25.673022 27966 catalog_manager.cc:1347] Generated new cluster ID: 97bc9237fa9240f3a975bb37de553910
I20260810 00:50:25.673149 27966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:50:25.714169 27966 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:50:25.714648 27966 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:50:25.726704 27966 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2bdfd6f97ee4f02ab61cdcc6c32d44b: Generated new TSK 0
I20260810 00:50:25.727627 27966 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 00:50:25.732250 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 27824
W20260810 00:50:25.733448 27935 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51747
W20260810 00:50:25.734134 28696 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 27824: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 00:50:25.734279 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 27824
2026-08-10T00:50:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2141 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:50:25 dist-test-slave-1nl6 krb5kdc[27993](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:50:25 dist-test-slave-1nl6 krb5kdc[27993](info): set up 2 sockets
Aug 10 00:50:25 dist-test-slave-1nl6 krb5kdc[27993](info): commencing operation
krb5kdc: starting...
W20260810 00:50:27.046139 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.265s user 0.005s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:50:27 dist-test-slave-1nl6 krb5kdc[27993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:50:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:50:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 00:50:27.323758 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34611
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34611
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:50:27.666311 28013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:50:27.666633 28013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:50:27.666808 28013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:50:27.673911 28013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:50:27.674057 28013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:50:27.674183 28013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:50:27.674290 28013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:50:27.674387 28013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:50:27.683964 28013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:50:27.685211 28013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:50:27.686501 28013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:50:27.698764 28019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:27.698787 28025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:27.698937 28023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:50:27.700071 28013 server_base.cc:1034] running on GCE node
I20260810 00:50:27.700786 28013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:50:27.702315 28013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:50:27.703531 28013 hybrid_clock.cc:648] HybridClock initialized: now 1786323027703470 us; error 51 us; skew 500 ppm
I20260810 00:50:27.703922 28013 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:50:27 dist-test-slave-1nl6 krb5kdc[27993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:50:27.707309 28013 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 00:50:27.708949 28013 webserver.cc:460] Webserver started at http://127.0.0.1:39411/ using document root <none> and password file <none>
I20260810 00:50:27.709637 28013 fs_manager.cc:359] Metadata directory not provided
I20260810 00:50:27.709787 28013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:50:27.710117 28013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:50:27.713259 28013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "8d3052db92a0416ea86f16e45b98e62c"
format_stamp: "Formatted at 2026-08-10 00:50:27 on dist-test-slave-1nl6"
server_key: "ade47cf7cfe56a53bb82f5a04e6f9de0"
server_key_iv: "d64315dc93f7d2a8a1e9e6c3f275fe43"
server_key_version: "encryptionkey@0"
I20260810 00:50:27.714057 28013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "8d3052db92a0416ea86f16e45b98e62c"
format_stamp: "Formatted at 2026-08-10 00:50:27 on dist-test-slave-1nl6"
server_key: "ade47cf7cfe56a53bb82f5a04e6f9de0"
server_key_iv: "d64315dc93f7d2a8a1e9e6c3f275fe43"
server_key_version: "encryptionkey@0"
I20260810 00:50:27.719300 28013 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 00:50:27.722625 28037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:27.723979 28013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 00:50:27.724169 28013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "8d3052db92a0416ea86f16e45b98e62c"
format_stamp: "Formatted at 2026-08-10 00:50:27 on dist-test-slave-1nl6"
server_key: "ade47cf7cfe56a53bb82f5a04e6f9de0"
server_key_iv: "d64315dc93f7d2a8a1e9e6c3f275fe43"
server_key_version: "encryptionkey@0"
I20260810 00:50:27.724377 28013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:50:27.769173 28013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:50:27.773638 28013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:50:27.773955 28013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:50:27.792755 28013 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34611
I20260810 00:50:27.792793 28117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34611 every 8 connection(s)
I20260810 00:50:27.794062 28013 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:50:27.797616 28118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:50:27.802639 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 28013
I20260810 00:50:27.802822 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:50:27.803382 28696 external_mini_cluster.cc:1428] Setting key 87ce56dde5cf407991a8df8a6445b7ca
I20260810 00:50:27.808622 28118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d3052db92a0416ea86f16e45b98e62c: Bootstrap starting.
I20260810 00:50:27.811650 28118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d3052db92a0416ea86f16e45b98e62c: Neither blocks nor log segments found. Creating new log.
I20260810 00:50:27.812614 28118 log.cc:824] T 00000000000000000000000000000000 P 8d3052db92a0416ea86f16e45b98e62c: Log is configured to *not* fsync() on all Append() calls
I20260810 00:50:27.815388 28118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d3052db92a0416ea86f16e45b98e62c: No bootstrap required, opened a new log
Aug 10 00:50:27 dist-test-slave-1nl6 krb5kdc[27993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 00:50:27.823912 28118 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d3052db92a0416ea86f16e45b98e62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3052db92a0416ea86f16e45b98e62c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34611 } }
I20260810 00:50:27.824254 28118 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d3052db92a0416ea86f16e45b98e62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:50:27.824383 28118 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d3052db92a0416ea86f16e45b98e62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d3052db92a0416ea86f16e45b98e62c, State: Initialized, Role: FOLLOWER
I20260810 00:50:27.824988 28118 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d3052db92a0416ea86f16e45b98e62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3052db92a0416ea86f16e45b98e62c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34611 } }
I20260810 00:50:27.825243 28118 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d3052db92a0416ea86f16e45b98e62c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:50:27.825398 28118 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d3052db92a0416ea86f16e45b98e62c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:50:27.825578 28118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d3052db92a0416ea86f16e45b98e62c [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:50:27.826642 28126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:27.805296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48877 (local address 127.0.0.1:34611)
0810 00:50:27.806103 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:27.806114 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:27.806158 (+ 44us) server_negotiation.cc:407] Connection header received
0810 00:50:27.807897 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:27.807920 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:27.808286 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:27.808676 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:27.810195 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:27.811538 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:27.812917 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:27.814040 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:27.814233 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:27.818715 (+ 4482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:27.818723 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:27.818737 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:50:27.818770 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:27.822229 (+ 3459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:27.823105 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:27.823111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:27.823117 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:27.823281 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:27.823705 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:27.823708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:27.823710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:27.824192 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:27.824530 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:27.824980 (+ 450us) server_negotiation.cc:299] Negotiation successful
0810 00:50:27.825609 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":353,"threads_started":1}
I20260810 00:50:27.828368 28118 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d3052db92a0416ea86f16e45b98e62c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3052db92a0416ea86f16e45b98e62c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34611 } }
I20260810 00:50:27.828835 28118 leader_election.cc:302] T 00000000000000000000000000000000 P 8d3052db92a0416ea86f16e45b98e62c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d3052db92a0416ea86f16e45b98e62c; no voters:
I20260810 00:50:27.829654 28118 leader_election.cc:288] T 00000000000000000000000000000000 P 8d3052db92a0416ea86f16e45b98e62c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:50:27.829759 28128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d3052db92a0416ea86f16e45b98e62c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:50:27.830694 28696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 00:50:27.830827 28128 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d3052db92a0416ea86f16e45b98e62c [term 1 LEADER]: Becoming Leader. State: Replica: 8d3052db92a0416ea86f16e45b98e62c, State: Running, Role: LEADER
I20260810 00:50:27.831418 28128 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d3052db92a0416ea86f16e45b98e62c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3052db92a0416ea86f16e45b98e62c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34611 } }
I20260810 00:50:27.831980 28118 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d3052db92a0416ea86f16e45b98e62c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:50:27.836071 28129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d3052db92a0416ea86f16e45b98e62c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d3052db92a0416ea86f16e45b98e62c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3052db92a0416ea86f16e45b98e62c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34611 } } }
I20260810 00:50:27.836534 28129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d3052db92a0416ea86f16e45b98e62c [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:27.836328 28130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d3052db92a0416ea86f16e45b98e62c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d3052db92a0416ea86f16e45b98e62c. Latest consensus state: current_term: 1 leader_uuid: "8d3052db92a0416ea86f16e45b98e62c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3052db92a0416ea86f16e45b98e62c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34611 } } }
I20260810 00:50:27.836680 28130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d3052db92a0416ea86f16e45b98e62c [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:27.843132 28141 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:50:27.848614 28141 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:50:27.859807 28141 catalog_manager.cc:1347] Generated new cluster ID: 366fb100b7f44ba3adb9f1022c2e892e
I20260810 00:50:27.859963 28141 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:50:27.870886 28126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:27.845808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48359 (local address 127.0.0.1:34611)
0810 00:50:27.845971 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:27.845979 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:27.857103 (+ 11124us) server_negotiation.cc:407] Connection header received
0810 00:50:27.857325 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:27.857331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:27.857430 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:27.857543 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:27.859760 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:27.860673 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:27.861932 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:27.862404 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:27.862547 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:27.864971 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:27.864979 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:27.864982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:50:27.865015 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:27.868004 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:27.868708 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:27.868720 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:27.868723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:27.868862 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:27.869279 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:27.869283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:27.869285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:27.869622 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:27.869900 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:27.870243 (+ 343us) server_negotiation.cc:299] Negotiation successful
0810 00:50:27.870493 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":56}
I20260810 00:50:27.883224 28141 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:50:27.884457 28141 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:50:27.896566 28141 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d3052db92a0416ea86f16e45b98e62c: Generated new TSK 0
I20260810 00:50:27.897308 28141 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 00:50:27.905830 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 28013
I20260810 00:50:27.906803 28112 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:50:27.998086 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 28013
2026-08-10T00:50:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2261 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:50:28 dist-test-slave-1nl6 krb5kdc[28164](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:50:28 dist-test-slave-1nl6 krb5kdc[28164](info): set up 2 sockets
Aug 10 00:50:28 dist-test-slave-1nl6 krb5kdc[28164](info): commencing operation
krb5kdc: starting...
W20260810 00:50:29.300231 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.258s user 0.000s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:50:29 dist-test-slave-1nl6 krb5kdc[28164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:50:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:50:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 00:50:29.591686 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33721
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:50:29.923301 28190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:50:29.923557 28190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:50:29.923746 28190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:50:29.930809 28190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:50:29.930950 28190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:50:29.931056 28190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:50:29.931224 28190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:50:29.931301 28190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:50:29.940938 28190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:50:29.942142 28190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:50:29.943464 28190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:50:29.955588 28196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:29.955662 28200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:29.955670 28195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:50:29.956378 28190 server_base.cc:1034] running on GCE node
I20260810 00:50:29.957104 28190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:50:29.958492 28190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:50:29.959720 28190 hybrid_clock.cc:648] HybridClock initialized: now 1786323029959675 us; error 57 us; skew 500 ppm
I20260810 00:50:29.960069 28190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:50:29 dist-test-slave-1nl6 krb5kdc[28164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:50:29.968794 28190 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 00:50:29.970280 28190 webserver.cc:460] Webserver started at http://127.0.0.1:37453/ using document root <none> and password file <none>
I20260810 00:50:29.970887 28190 fs_manager.cc:359] Metadata directory not provided
I20260810 00:50:29.971063 28190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:50:29.971320 28190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:50:29.974222 28190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "5989c46731cd4876aa73d2eada4cbec9"
format_stamp: "Formatted at 2026-08-10 00:50:29 on dist-test-slave-1nl6"
server_key: "1ff39b354ca16521dcc82f5834e3b149"
server_key_iv: "4ac75509a779c70419376c4be7cb7db6"
server_key_version: "encryptionkey@0"
I20260810 00:50:29.975003 28190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "5989c46731cd4876aa73d2eada4cbec9"
format_stamp: "Formatted at 2026-08-10 00:50:29 on dist-test-slave-1nl6"
server_key: "1ff39b354ca16521dcc82f5834e3b149"
server_key_iv: "4ac75509a779c70419376c4be7cb7db6"
server_key_version: "encryptionkey@0"
I20260810 00:50:29.979902 28190 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 00:50:29.983143 28212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:29.984495 28190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 00:50:29.984676 28190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "5989c46731cd4876aa73d2eada4cbec9"
format_stamp: "Formatted at 2026-08-10 00:50:29 on dist-test-slave-1nl6"
server_key: "1ff39b354ca16521dcc82f5834e3b149"
server_key_iv: "4ac75509a779c70419376c4be7cb7db6"
server_key_version: "encryptionkey@0"
I20260810 00:50:29.984891 28190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:50:30.015818 28190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:50:30.020449 28190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:50:30.020748 28190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:50:30.039673 28190 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33721
I20260810 00:50:30.039713 28289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33721 every 8 connection(s)
I20260810 00:50:30.041173 28190 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:50:30.044965 28292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:50:30.048487 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 28190
I20260810 00:50:30.048674 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:50:30.049275 28696 external_mini_cluster.cc:1428] Setting key 35d9b11f668b4f0bf6e205721ec99b63
I20260810 00:50:30.061636 28292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5989c46731cd4876aa73d2eada4cbec9: Bootstrap starting.
I20260810 00:50:30.066088 28292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5989c46731cd4876aa73d2eada4cbec9: Neither blocks nor log segments found. Creating new log.
I20260810 00:50:30.067324 28297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:30.052130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41097 (local address 127.0.0.1:33721)
0810 00:50:30.053036 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:30.053051 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:30.053107 (+ 56us) server_negotiation.cc:407] Connection header received
0810 00:50:30.055800 (+ 2693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:30.055829 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:30.056285 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:30.057325 (+ 1040us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:30.057793 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:30.059800 (+ 2007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:30.061593 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:30.062583 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:30.062786 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:30.064070 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:30.064078 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:30.064091 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:30.064937 (+ 846us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:30.065053 (+ 116us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:30.065089 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:30.065235 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:30.065474 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 00:50:30.065745 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":362,"threads_started":1}
I20260810 00:50:30.067571 28292 log.cc:824] T 00000000000000000000000000000000 P 5989c46731cd4876aa73d2eada4cbec9: Log is configured to *not* fsync() on all Append() calls
I20260810 00:50:30.071473 28696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 00:50:30.071702 28292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5989c46731cd4876aa73d2eada4cbec9: No bootstrap required, opened a new log
I20260810 00:50:30.083482 28297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:30.076941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36865 (local address 127.0.0.1:33721)
0810 00:50:30.077161 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:30.077170 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:30.077755 (+ 585us) server_negotiation.cc:407] Connection header received
0810 00:50:30.077970 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:30.077975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:30.078052 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:30.078162 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:30.078638 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:30.079436 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:30.080764 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:30.081225 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:30.081388 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:30.082588 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:30.082595 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:30.082599 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:30.082622 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:30.082670 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:30.082691 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:30.082778 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:30.083063 (+ 285us) server_negotiation.cc:299] Negotiation successful
0810 00:50:30.083164 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 00:50:30.083423 28292 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5989c46731cd4876aa73d2eada4cbec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5989c46731cd4876aa73d2eada4cbec9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33721 } }
I20260810 00:50:30.083897 28292 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5989c46731cd4876aa73d2eada4cbec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:50:30.084026 28292 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5989c46731cd4876aa73d2eada4cbec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5989c46731cd4876aa73d2eada4cbec9, State: Initialized, Role: FOLLOWER
I20260810 00:50:30.084751 28292 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5989c46731cd4876aa73d2eada4cbec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5989c46731cd4876aa73d2eada4cbec9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33721 } }
I20260810 00:50:30.085088 28292 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5989c46731cd4876aa73d2eada4cbec9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:50:30.085256 28292 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5989c46731cd4876aa73d2eada4cbec9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:50:30.085457 28292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5989c46731cd4876aa73d2eada4cbec9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:50:30.088699 28292 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5989c46731cd4876aa73d2eada4cbec9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5989c46731cd4876aa73d2eada4cbec9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33721 } }
I20260810 00:50:30.089355 28292 leader_election.cc:302] T 00000000000000000000000000000000 P 5989c46731cd4876aa73d2eada4cbec9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5989c46731cd4876aa73d2eada4cbec9; no voters:
I20260810 00:50:30.090091 28292 leader_election.cc:288] T 00000000000000000000000000000000 P 5989c46731cd4876aa73d2eada4cbec9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:50:30.090142 28306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5989c46731cd4876aa73d2eada4cbec9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:50:30.090973 28306 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5989c46731cd4876aa73d2eada4cbec9 [term 1 LEADER]: Becoming Leader. State: Replica: 5989c46731cd4876aa73d2eada4cbec9, State: Running, Role: LEADER
I20260810 00:50:30.091612 28306 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5989c46731cd4876aa73d2eada4cbec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5989c46731cd4876aa73d2eada4cbec9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33721 } }
I20260810 00:50:30.092543 28292 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5989c46731cd4876aa73d2eada4cbec9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:50:30.095350 28311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5989c46731cd4876aa73d2eada4cbec9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5989c46731cd4876aa73d2eada4cbec9. Latest consensus state: current_term: 1 leader_uuid: "5989c46731cd4876aa73d2eada4cbec9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5989c46731cd4876aa73d2eada4cbec9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33721 } } }
I20260810 00:50:30.095815 28311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5989c46731cd4876aa73d2eada4cbec9 [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:30.096395 28307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5989c46731cd4876aa73d2eada4cbec9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5989c46731cd4876aa73d2eada4cbec9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5989c46731cd4876aa73d2eada4cbec9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33721 } } }
I20260810 00:50:30.096698 28307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5989c46731cd4876aa73d2eada4cbec9 [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:30.096724 28320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:50:30.102499 28320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:50:30.114660 28320 catalog_manager.cc:1347] Generated new cluster ID: 8b7bd395f9194820839ace3f6d00486a
I20260810 00:50:30.114795 28320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:50:30.128221 28320 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:50:30.129462 28320 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:50:30.141812 28320 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5989c46731cd4876aa73d2eada4cbec9: Generated new TSK 0
I20260810 00:50:30.142918 28320 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 00:50:30.161434 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 28190
W20260810 00:50:30.162635 28280 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41097
W20260810 00:50:30.163507 28696 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 28190: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 00:50:30.163666 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 28190
2026-08-10T00:50:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2165 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:50:30 dist-test-slave-1nl6 krb5kdc[28347](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:50:30 dist-test-slave-1nl6 krb5kdc[28347](info): set up 2 sockets
Aug 10 00:50:30 dist-test-slave-1nl6 krb5kdc[28347](info): commencing operation
krb5kdc: starting...
W20260810 00:50:31.472010 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.264s user 0.000s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:50:31 dist-test-slave-1nl6 krb5kdc[28347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:50:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:50:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 00:50:31.760047 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45273
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:50:32.089148 28365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:50:32.089422 28365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:50:32.089545 28365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:50:32.096608 28365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:50:32.096706 28365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:50:32.096756 28365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:50:32.096841 28365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:50:32.096911 28365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:50:32.106863 28365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:50:32.108057 28365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:50:32.109378 28365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:50:32.121188 28373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:32.121183 28378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:32.121225 28374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:50:32.122216 28365 server_base.cc:1034] running on GCE node
I20260810 00:50:32.123072 28365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:50:32.124418 28365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:50:32.125639 28365 hybrid_clock.cc:648] HybridClock initialized: now 1786323032125603 us; error 36 us; skew 500 ppm
I20260810 00:50:32.125991 28365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:50:32 dist-test-slave-1nl6 krb5kdc[28347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:50:32.129052 28365 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 00:50:32.130582 28365 webserver.cc:460] Webserver started at http://127.0.0.1:36209/ using document root <none> and password file <none>
I20260810 00:50:32.131312 28365 fs_manager.cc:359] Metadata directory not provided
I20260810 00:50:32.131461 28365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:50:32.131773 28365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:50:32.134713 28365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "c2eef44c134948deb478054afac1a3a9"
format_stamp: "Formatted at 2026-08-10 00:50:32 on dist-test-slave-1nl6"
server_key: "6f34a6c72457de18afbacb7e0bad32a2"
server_key_iv: "4709100ddfa970088d288e4640adaa89"
server_key_version: "encryptionkey@0"
I20260810 00:50:32.135572 28365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "c2eef44c134948deb478054afac1a3a9"
format_stamp: "Formatted at 2026-08-10 00:50:32 on dist-test-slave-1nl6"
server_key: "6f34a6c72457de18afbacb7e0bad32a2"
server_key_iv: "4709100ddfa970088d288e4640adaa89"
server_key_version: "encryptionkey@0"
I20260810 00:50:32.142519 28365 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.004s
I20260810 00:50:32.147167 28386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:32.148423 28365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 00:50:32.148624 28365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "c2eef44c134948deb478054afac1a3a9"
format_stamp: "Formatted at 2026-08-10 00:50:32 on dist-test-slave-1nl6"
server_key: "6f34a6c72457de18afbacb7e0bad32a2"
server_key_iv: "4709100ddfa970088d288e4640adaa89"
server_key_version: "encryptionkey@0"
I20260810 00:50:32.148823 28365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:50:32.154327 28365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:50:32.158094 28365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:50:32.158382 28365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:50:32.176558 28365 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45273
I20260810 00:50:32.176635 28462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45273 every 8 connection(s)
I20260810 00:50:32.177838 28365 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:50:32.181476 28463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:50:32.186997 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 28365
I20260810 00:50:32.187206 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:50:32.187785 28696 external_mini_cluster.cc:1428] Setting key 451e8ced0e7df4328590e15421871888
I20260810 00:50:32.192549 28463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2eef44c134948deb478054afac1a3a9: Bootstrap starting.
I20260810 00:50:32.195562 28467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:32.189534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51663 (local address 127.0.0.1:45273)
0810 00:50:32.190358 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:32.190387 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:32.190430 (+ 43us) server_negotiation.cc:407] Connection header received
0810 00:50:32.192242 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:32.192264 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:32.192567 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:32.192920 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:32.193991 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:32.193996 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:32.194008 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:32.194021 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:32.194076 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:32.194097 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:32.194262 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:32.194407 (+ 145us) server_negotiation.cc:299] Negotiation successful
0810 00:50:32.194597 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":395,"threads_started":1}
I20260810 00:50:32.196197 28463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2eef44c134948deb478054afac1a3a9: Neither blocks nor log segments found. Creating new log.
I20260810 00:50:32.197460 28463 log.cc:824] T 00000000000000000000000000000000 P c2eef44c134948deb478054afac1a3a9: Log is configured to *not* fsync() on all Append() calls
I20260810 00:50:32.199221 28696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 00:50:32.200368 28463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2eef44c134948deb478054afac1a3a9: No bootstrap required, opened a new log
I20260810 00:50:32.206001 28467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:32.203585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35329 (local address 127.0.0.1:45273)
0810 00:50:32.203790 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:32.203796 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:32.204151 (+ 355us) server_negotiation.cc:407] Connection header received
0810 00:50:32.204311 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:32.204314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:32.204358 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:32.204436 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:32.205355 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:32.205359 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:32.205362 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:32.205379 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:32.205413 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:32.205427 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:32.205522 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:32.205655 (+ 133us) server_negotiation.cc:299] Negotiation successful
0810 00:50:32.205698 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 00:50:32.208845 28463 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2eef44c134948deb478054afac1a3a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2eef44c134948deb478054afac1a3a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45273 } }
I20260810 00:50:32.209193 28463 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2eef44c134948deb478054afac1a3a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:50:32.209301 28463 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2eef44c134948deb478054afac1a3a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2eef44c134948deb478054afac1a3a9, State: Initialized, Role: FOLLOWER
I20260810 00:50:32.209892 28463 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2eef44c134948deb478054afac1a3a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2eef44c134948deb478054afac1a3a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45273 } }
I20260810 00:50:32.210152 28463 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2eef44c134948deb478054afac1a3a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:50:32.210294 28463 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2eef44c134948deb478054afac1a3a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:50:32.210501 28463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2eef44c134948deb478054afac1a3a9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:50:32.212778 28463 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2eef44c134948deb478054afac1a3a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2eef44c134948deb478054afac1a3a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45273 } }
I20260810 00:50:32.213291 28463 leader_election.cc:302] T 00000000000000000000000000000000 P c2eef44c134948deb478054afac1a3a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2eef44c134948deb478054afac1a3a9; no voters:
I20260810 00:50:32.213831 28463 leader_election.cc:288] T 00000000000000000000000000000000 P c2eef44c134948deb478054afac1a3a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:50:32.213958 28480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2eef44c134948deb478054afac1a3a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:50:32.214660 28480 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2eef44c134948deb478054afac1a3a9 [term 1 LEADER]: Becoming Leader. State: Replica: c2eef44c134948deb478054afac1a3a9, State: Running, Role: LEADER
I20260810 00:50:32.215207 28480 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2eef44c134948deb478054afac1a3a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2eef44c134948deb478054afac1a3a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45273 } }
I20260810 00:50:32.215883 28463 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2eef44c134948deb478054afac1a3a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:50:32.218961 28485 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2eef44c134948deb478054afac1a3a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2eef44c134948deb478054afac1a3a9. Latest consensus state: current_term: 1 leader_uuid: "c2eef44c134948deb478054afac1a3a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2eef44c134948deb478054afac1a3a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45273 } } }
I20260810 00:50:32.219432 28485 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2eef44c134948deb478054afac1a3a9 [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:32.220391 28495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:50:32.220232 28481 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2eef44c134948deb478054afac1a3a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2eef44c134948deb478054afac1a3a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2eef44c134948deb478054afac1a3a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45273 } } }
I20260810 00:50:32.220607 28481 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2eef44c134948deb478054afac1a3a9 [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:32.224841 28495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:50:32.233731 28495 catalog_manager.cc:1347] Generated new cluster ID: a41f3e0fc775471797bc953054dc7181
I20260810 00:50:32.233840 28495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:50:32.267467 28495 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:50:32.267897 28495 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:50:32.278177 28495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2eef44c134948deb478054afac1a3a9: Generated new TSK 0
I20260810 00:50:32.278972 28495 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 00:50:32.286223 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 28365
W20260810 00:50:32.287245 28457 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51663
W20260810 00:50:32.287889 28696 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 28365: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 00:50:32.288041 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 28365
2026-08-10T00:50:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2126 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:50:32 dist-test-slave-1nl6 krb5kdc[28515](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:50:32 dist-test-slave-1nl6 krb5kdc[28515](info): set up 2 sockets
Aug 10 00:50:32 dist-test-slave-1nl6 krb5kdc[28515](info): commencing operation
krb5kdc: starting...
W20260810 00:50:33.612597 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.277s user 0.000s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:50:33 dist-test-slave-1nl6 krb5kdc[28515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:50:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:50:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 00:50:33.905237 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40151
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:50:34.244772 28535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:50:34.245107 28535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:50:34.245292 28535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:50:34.252278 28535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:50:34.252424 28535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:50:34.252532 28535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:50:34.252646 28535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:50:34.252717 28535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:50:34.262094 28535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:50:34.263381 28535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:50:34.264686 28535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:50:34.276643 28543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:34.276667 28546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:34.276805 28544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:50:34.278491 28535 server_base.cc:1034] running on GCE node
I20260810 00:50:34.279260 28535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:50:34.280519 28535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:50:34.281759 28535 hybrid_clock.cc:648] HybridClock initialized: now 1786323034281711 us; error 40 us; skew 500 ppm
I20260810 00:50:34.282136 28535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:50:34 dist-test-slave-1nl6 krb5kdc[28515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:50:34.285387 28535 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 00:50:34.286872 28535 webserver.cc:460] Webserver started at http://127.0.0.1:40567/ using document root <none> and password file <none>
I20260810 00:50:34.287608 28535 fs_manager.cc:359] Metadata directory not provided
I20260810 00:50:34.287747 28535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:50:34.288064 28535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:50:34.290990 28535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "9e79a36b018749e59bc19819d0995a85"
format_stamp: "Formatted at 2026-08-10 00:50:34 on dist-test-slave-1nl6"
server_key: "34bb3416f4cd279f36bcb6b94b7a5654"
server_key_iv: "aaa3a3d87a0a0e0cda3209d501c6fbf3"
server_key_version: "encryptionkey@0"
I20260810 00:50:34.291787 28535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "9e79a36b018749e59bc19819d0995a85"
format_stamp: "Formatted at 2026-08-10 00:50:34 on dist-test-slave-1nl6"
server_key: "34bb3416f4cd279f36bcb6b94b7a5654"
server_key_iv: "aaa3a3d87a0a0e0cda3209d501c6fbf3"
server_key_version: "encryptionkey@0"
I20260810 00:50:34.297019 28535 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 00:50:34.300371 28554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:34.301607 28535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 00:50:34.301793 28535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "9e79a36b018749e59bc19819d0995a85"
format_stamp: "Formatted at 2026-08-10 00:50:34 on dist-test-slave-1nl6"
server_key: "34bb3416f4cd279f36bcb6b94b7a5654"
server_key_iv: "aaa3a3d87a0a0e0cda3209d501c6fbf3"
server_key_version: "encryptionkey@0"
I20260810 00:50:34.301986 28535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:50:34.314858 28535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:50:34.319309 28535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:50:34.319600 28535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:50:34.338313 28535 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40151
I20260810 00:50:34.338359 28636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40151 every 8 connection(s)
I20260810 00:50:34.339713 28535 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:50:34.342252 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 28535
I20260810 00:50:34.342437 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:50:34.343024 28696 external_mini_cluster.cc:1428] Setting key 1e911e3cdee70db51c969c9361507c7e
I20260810 00:50:34.343783 28637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 00:50:34 dist-test-slave-1nl6 krb5kdc[28515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 00:50:34.358259 28637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e79a36b018749e59bc19819d0995a85: Bootstrap starting.
I20260810 00:50:34.361748 28637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e79a36b018749e59bc19819d0995a85: Neither blocks nor log segments found. Creating new log.
I20260810 00:50:34.362762 28637 log.cc:824] T 00000000000000000000000000000000 P 9e79a36b018749e59bc19819d0995a85: Log is configured to *not* fsync() on all Append() calls
I20260810 00:50:34.365696 28637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e79a36b018749e59bc19819d0995a85: No bootstrap required, opened a new log
I20260810 00:50:34.366148 28643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:34.344870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:59997 (local address 127.0.0.1:40151)
0810 00:50:34.345633 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:34.345644 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:34.345684 (+ 40us) server_negotiation.cc:407] Connection header received
0810 00:50:34.347292 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:34.347313 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:34.347698 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:34.348013 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:34.349817 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:34.351188 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:34.352993 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:34.353621 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:34.353745 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:34.358657 (+ 4912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:34.358680 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:34.358698 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:50:34.358756 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:34.361882 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:34.362676 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:34.362683 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:34.362689 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:34.362866 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:34.363426 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:34.363431 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:34.363441 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:34.363973 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:34.364307 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:34.364818 (+ 511us) server_negotiation.cc:299] Negotiation successful
0810 00:50:34.365131 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":335,"threads_started":1}
I20260810 00:50:34.370296 28696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 00:50:34.374347 28637 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e79a36b018749e59bc19819d0995a85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e79a36b018749e59bc19819d0995a85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40151 } }
I20260810 00:50:34.374840 28637 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e79a36b018749e59bc19819d0995a85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:50:34.374996 28637 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e79a36b018749e59bc19819d0995a85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e79a36b018749e59bc19819d0995a85, State: Initialized, Role: FOLLOWER
I20260810 00:50:34.375728 28637 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e79a36b018749e59bc19819d0995a85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e79a36b018749e59bc19819d0995a85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40151 } }
I20260810 00:50:34.375982 28637 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e79a36b018749e59bc19819d0995a85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:50:34.376143 28637 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e79a36b018749e59bc19819d0995a85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:50:34.376331 28637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e79a36b018749e59bc19819d0995a85 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:50:34.378535 28637 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e79a36b018749e59bc19819d0995a85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e79a36b018749e59bc19819d0995a85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40151 } }
I20260810 00:50:34.379155 28637 leader_election.cc:302] T 00000000000000000000000000000000 P 9e79a36b018749e59bc19819d0995a85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e79a36b018749e59bc19819d0995a85; no voters:
I20260810 00:50:34.379752 28637 leader_election.cc:288] T 00000000000000000000000000000000 P 9e79a36b018749e59bc19819d0995a85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:50:34.379921 28650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e79a36b018749e59bc19819d0995a85 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:50:34.381423 28650 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e79a36b018749e59bc19819d0995a85 [term 1 LEADER]: Becoming Leader. State: Replica: 9e79a36b018749e59bc19819d0995a85, State: Running, Role: LEADER
I20260810 00:50:34.382186 28637 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e79a36b018749e59bc19819d0995a85 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:50:34.382093 28650 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e79a36b018749e59bc19819d0995a85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e79a36b018749e59bc19819d0995a85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40151 } }
I20260810 00:50:34.384737 28652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e79a36b018749e59bc19819d0995a85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e79a36b018749e59bc19819d0995a85. Latest consensus state: current_term: 1 leader_uuid: "9e79a36b018749e59bc19819d0995a85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e79a36b018749e59bc19819d0995a85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40151 } } }
I20260810 00:50:34.385161 28652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e79a36b018749e59bc19819d0995a85 [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:34.385720 28651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e79a36b018749e59bc19819d0995a85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e79a36b018749e59bc19819d0995a85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e79a36b018749e59bc19819d0995a85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40151 } } }
I20260810 00:50:34.386013 28651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e79a36b018749e59bc19819d0995a85 [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:34.394301 28658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:50:34.395352 28643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:34.374880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:49625 (local address 127.0.0.1:40151)
0810 00:50:34.375145 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:34.375154 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:34.375468 (+ 314us) server_negotiation.cc:407] Connection header received
0810 00:50:34.375669 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:34.375674 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:34.375738 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:34.375841 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:34.377464 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:34.378227 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:34.379888 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:34.380687 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:34.380822 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:34.384689 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:34.384698 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:34.384706 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:50:34.384777 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:34.389664 (+ 4887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:34.391049 (+ 1385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:34.391057 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:34.391064 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:34.391293 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:34.392034 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:34.392041 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:34.392047 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:34.392557 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:34.394330 (+ 1773us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:34.394565 (+ 235us) server_negotiation.cc:299] Negotiation successful
0810 00:50:34.394810 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260810 00:50:34.399955 28658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:50:34.410259 28658 catalog_manager.cc:1347] Generated new cluster ID: d71cdb0397404d70a8791062a35b47e7
I20260810 00:50:34.410372 28658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:50:34.430106 28658 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:50:34.431495 28658 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:50:34.439731 28658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e79a36b018749e59bc19819d0995a85: Generated new TSK 0
I20260810 00:50:34.440603 28658 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 00:50:34.456177 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 28535
I20260810 00:50:34.457341 28628 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:50:34.548338 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 28535
W20260810 00:50:34.572618 28512 connection.cc:535] client connection to 127.0.0.1:40151 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40151): Connection reset by peer (error 104)
2026-08-10T00:50:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2257 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:50:34 dist-test-slave-1nl6 krb5kdc[28688](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:50:34 dist-test-slave-1nl6 krb5kdc[28688](info): set up 2 sockets
Aug 10 00:50:34 dist-test-slave-1nl6 krb5kdc[28688](info): commencing operation
krb5kdc: starting...
W20260810 00:50:35.879297 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.286s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:50:36 dist-test-slave-1nl6 krb5kdc[28688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:50:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:50:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 00:50:36.197717 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45705
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:50:36.562597 28711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:50:36.562963 28711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:50:36.563062 28711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:50:36.570510 28711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:50:36.570647 28711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:50:36.570710 28711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:50:36.570776 28711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:50:36.570884 28711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:50:36.581542 28711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:50:36.582968 28711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:50:36.584333 28711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:50:36.597599 28716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:36.597637 28718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:36.597549 28720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:50:36.600457 28711 server_base.cc:1034] running on GCE node
I20260810 00:50:36.601330 28711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:50:36.602732 28711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:50:36.603986 28711 hybrid_clock.cc:648] HybridClock initialized: now 1786323036603946 us; error 47 us; skew 500 ppm
I20260810 00:50:36.604362 28711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:50:36 dist-test-slave-1nl6 krb5kdc[28688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:50:36.611969 28711 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 00:50:36.613631 28711 webserver.cc:460] Webserver started at http://127.0.0.1:42195/ using document root <none> and password file <none>
I20260810 00:50:36.614348 28711 fs_manager.cc:359] Metadata directory not provided
I20260810 00:50:36.614487 28711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:50:36.614817 28711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:50:36.617928 28711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "6063c0d5cae04a82a20e885b7cedede1"
format_stamp: "Formatted at 2026-08-10 00:50:36 on dist-test-slave-1nl6"
server_key: "6324091eadabc14819c263238f220fa9"
server_key_iv: "6d5ed3f5a6712ba7ab7045e2bdb5e206"
server_key_version: "encryptionkey@0"
I20260810 00:50:36.618732 28711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "6063c0d5cae04a82a20e885b7cedede1"
format_stamp: "Formatted at 2026-08-10 00:50:36 on dist-test-slave-1nl6"
server_key: "6324091eadabc14819c263238f220fa9"
server_key_iv: "6d5ed3f5a6712ba7ab7045e2bdb5e206"
server_key_version: "encryptionkey@0"
I20260810 00:50:36.624246 28711 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 00:50:36.627785 28733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:36.629117 28711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 00:50:36.629323 28711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "6063c0d5cae04a82a20e885b7cedede1"
format_stamp: "Formatted at 2026-08-10 00:50:36 on dist-test-slave-1nl6"
server_key: "6324091eadabc14819c263238f220fa9"
server_key_iv: "6d5ed3f5a6712ba7ab7045e2bdb5e206"
server_key_version: "encryptionkey@0"
I20260810 00:50:36.629547 28711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:50:36.646556 28711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:50:36.651197 28711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:50:36.651539 28711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:50:36.671432 28711 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45705
I20260810 00:50:36.671470 28811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45705 every 8 connection(s)
I20260810 00:50:36.672950 28711 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:50:36.677016 28812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:50:36.683300 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 28711
I20260810 00:50:36.683511 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:50:36.684116 28696 external_mini_cluster.cc:1428] Setting key 490e23348781eb6233e84909a5082583
I20260810 00:50:36.689092 28812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6063c0d5cae04a82a20e885b7cedede1: Bootstrap starting.
I20260810 00:50:36.692471 28812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6063c0d5cae04a82a20e885b7cedede1: Neither blocks nor log segments found. Creating new log.
I20260810 00:50:36.693522 28812 log.cc:824] T 00000000000000000000000000000000 P 6063c0d5cae04a82a20e885b7cedede1: Log is configured to *not* fsync() on all Append() calls
I20260810 00:50:36.696703 28812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6063c0d5cae04a82a20e885b7cedede1: No bootstrap required, opened a new log
I20260810 00:50:36.698244 28817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:36.686186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:35199 (local address 127.0.0.1:45705)
0810 00:50:36.687282 (+ 1096us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:36.687295 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:36.687340 (+ 45us) server_negotiation.cc:407] Connection header received
0810 00:50:36.689085 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:36.689110 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:36.689602 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:36.690018 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:36.690626 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:36.692083 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:36.693682 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:36.694398 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:36.694616 (+ 218us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:36.695992 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:36.696000 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:36.696015 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:36.696554 (+ 539us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:36.696626 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:36.696651 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:36.696843 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:36.697121 (+ 278us) server_negotiation.cc:299] Negotiation successful
0810 00:50:36.697314 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":847,"thread_start_us":575,"threads_started":1}
I20260810 00:50:36.702580 28696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 00:50:36.705711 28812 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6063c0d5cae04a82a20e885b7cedede1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6063c0d5cae04a82a20e885b7cedede1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45705 } }
I20260810 00:50:36.706275 28812 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6063c0d5cae04a82a20e885b7cedede1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:50:36.706400 28812 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6063c0d5cae04a82a20e885b7cedede1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6063c0d5cae04a82a20e885b7cedede1, State: Initialized, Role: FOLLOWER
I20260810 00:50:36.707162 28812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6063c0d5cae04a82a20e885b7cedede1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6063c0d5cae04a82a20e885b7cedede1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45705 } }
I20260810 00:50:36.707625 28812 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6063c0d5cae04a82a20e885b7cedede1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:50:36.707787 28812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6063c0d5cae04a82a20e885b7cedede1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:50:36.708016 28812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6063c0d5cae04a82a20e885b7cedede1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:50:36.710394 28812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6063c0d5cae04a82a20e885b7cedede1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6063c0d5cae04a82a20e885b7cedede1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45705 } }
I20260810 00:50:36.710978 28812 leader_election.cc:302] T 00000000000000000000000000000000 P 6063c0d5cae04a82a20e885b7cedede1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6063c0d5cae04a82a20e885b7cedede1; no voters:
I20260810 00:50:36.711751 28812 leader_election.cc:288] T 00000000000000000000000000000000 P 6063c0d5cae04a82a20e885b7cedede1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:50:36.711875 28827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6063c0d5cae04a82a20e885b7cedede1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:50:36.712651 28827 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6063c0d5cae04a82a20e885b7cedede1 [term 1 LEADER]: Becoming Leader. State: Replica: 6063c0d5cae04a82a20e885b7cedede1, State: Running, Role: LEADER
I20260810 00:50:36.713764 28827 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6063c0d5cae04a82a20e885b7cedede1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6063c0d5cae04a82a20e885b7cedede1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45705 } }
I20260810 00:50:36.714329 28817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:36.707355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:57745 (local address 127.0.0.1:45705)
0810 00:50:36.707551 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:36.707560 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:36.708012 (+ 452us) server_negotiation.cc:407] Connection header received
0810 00:50:36.708238 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:36.708242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:36.708313 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:36.708433 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:36.708950 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:36.709756 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:36.711309 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:36.711879 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:36.712037 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:36.713450 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:36.713455 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:36.713460 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:36.713481 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:36.713528 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:36.713554 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:36.713691 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:36.713897 (+ 206us) server_negotiation.cc:299] Negotiation successful
0810 00:50:36.713979 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 00:50:36.714835 28812 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6063c0d5cae04a82a20e885b7cedede1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:50:36.721948 28829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6063c0d5cae04a82a20e885b7cedede1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6063c0d5cae04a82a20e885b7cedede1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6063c0d5cae04a82a20e885b7cedede1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45705 } } }
I20260810 00:50:36.722419 28829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6063c0d5cae04a82a20e885b7cedede1 [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:36.723031 28828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6063c0d5cae04a82a20e885b7cedede1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6063c0d5cae04a82a20e885b7cedede1. Latest consensus state: current_term: 1 leader_uuid: "6063c0d5cae04a82a20e885b7cedede1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6063c0d5cae04a82a20e885b7cedede1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45705 } } }
I20260810 00:50:36.723408 28828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6063c0d5cae04a82a20e885b7cedede1 [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:36.724282 28836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:50:36.730084 28836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:50:36.740111 28836 catalog_manager.cc:1347] Generated new cluster ID: 0e6c7947a4734f01b4fbeb798df348ac
I20260810 00:50:36.740238 28836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:50:36.773183 28836 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:50:36.774883 28836 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:50:36.785028 28836 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6063c0d5cae04a82a20e885b7cedede1: Generated new TSK 0
I20260810 00:50:36.786090 28836 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 00:50:36.800923 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 28711
W20260810 00:50:36.802258 28803 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.98:35199
W20260810 00:50:36.803169 28696 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 28711: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 00:50:36.803340 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 28711
W20260810 00:50:36.825474 28685 connection.cc:535] client connection to 127.0.0.1:45705 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45705): Connection reset by peer (error 104)
2026-08-10T00:50:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2251 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:50:36 dist-test-slave-1nl6 krb5kdc[28857](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:50:36 dist-test-slave-1nl6 krb5kdc[28857](info): set up 2 sockets
Aug 10 00:50:36 dist-test-slave-1nl6 krb5kdc[28857](info): commencing operation
krb5kdc: starting...
W20260810 00:50:38.126644 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.282s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:50:38 dist-test-slave-1nl6 krb5kdc[28857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:50:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:50:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 00:50:38.419304 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42083
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:50:38.758349 28876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:50:38.758613 28876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:50:38.758711 28876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:50:38.765861 28876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:50:38.765964 28876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:50:38.766072 28876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:50:38.766201 28876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:50:38.766276 28876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:50:38.775964 28876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:50:38.777264 28876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:50:38.778605 28876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:50:38.790441 28881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:38.790485 28882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:38.790748 28885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:50:38.791417 28876 server_base.cc:1034] running on GCE node
I20260810 00:50:38.792238 28876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:50:38.793677 28876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:50:38.794958 28876 hybrid_clock.cc:648] HybridClock initialized: now 1786323038794853 us; error 56 us; skew 500 ppm
I20260810 00:50:38.795341 28876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:50:38 dist-test-slave-1nl6 krb5kdc[28857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:50:38.798447 28876 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 00:50:38.799930 28876 webserver.cc:460] Webserver started at http://127.0.0.1:43505/ using document root <none> and password file <none>
I20260810 00:50:38.800595 28876 fs_manager.cc:359] Metadata directory not provided
I20260810 00:50:38.800745 28876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:50:38.801069 28876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:50:38.804098 28876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "fe6fe477f25f4f10a67a84ea0fce4c89"
format_stamp: "Formatted at 2026-08-10 00:50:38 on dist-test-slave-1nl6"
server_key: "25286ba4e8db3c0d1c5b2efa1b7fcbda"
server_key_iv: "dfc0c3609e1dc890433432cef2710c3c"
server_key_version: "encryptionkey@0"
I20260810 00:50:38.804880 28876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "fe6fe477f25f4f10a67a84ea0fce4c89"
format_stamp: "Formatted at 2026-08-10 00:50:38 on dist-test-slave-1nl6"
server_key: "25286ba4e8db3c0d1c5b2efa1b7fcbda"
server_key_iv: "dfc0c3609e1dc890433432cef2710c3c"
server_key_version: "encryptionkey@0"
I20260810 00:50:38.809911 28876 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260810 00:50:38.813277 28894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:38.814574 28876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 00:50:38.814772 28876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "fe6fe477f25f4f10a67a84ea0fce4c89"
format_stamp: "Formatted at 2026-08-10 00:50:38 on dist-test-slave-1nl6"
server_key: "25286ba4e8db3c0d1c5b2efa1b7fcbda"
server_key_iv: "dfc0c3609e1dc890433432cef2710c3c"
server_key_version: "encryptionkey@0"
I20260810 00:50:38.815042 28876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:50:38.820624 28876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:50:38.824617 28876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:50:38.824934 28876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:50:38.844369 28876 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42083
I20260810 00:50:38.844452 28973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42083 every 8 connection(s)
I20260810 00:50:38.845826 28876 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:50:38.849500 28974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:50:38.855346 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 28876
I20260810 00:50:38.855572 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:50:38.856148 28696 external_mini_cluster.cc:1428] Setting key 0f02418ec2f11627367104d03155e1f0
I20260810 00:50:38.860532 28974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe6fe477f25f4f10a67a84ea0fce4c89: Bootstrap starting.
I20260810 00:50:38.864123 28977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:38.858030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:51973 (local address 127.0.0.1:42083)
0810 00:50:38.858962 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:38.859010 (+ 48us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:38.859053 (+ 43us) server_negotiation.cc:407] Connection header received
0810 00:50:38.860702 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:38.860731 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:38.861062 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:38.861384 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:38.862510 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:38.862520 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:38.862532 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:38.862546 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:38.862611 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:38.862631 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:38.862802 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:38.863006 (+ 204us) server_negotiation.cc:299] Negotiation successful
0810 00:50:38.863252 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":417,"threads_started":1}
I20260810 00:50:38.864248 28974 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe6fe477f25f4f10a67a84ea0fce4c89: Neither blocks nor log segments found. Creating new log.
I20260810 00:50:38.865659 28974 log.cc:824] T 00000000000000000000000000000000 P fe6fe477f25f4f10a67a84ea0fce4c89: Log is configured to *not* fsync() on all Append() calls
I20260810 00:50:38.868283 28696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 00:50:38.868814 28974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe6fe477f25f4f10a67a84ea0fce4c89: No bootstrap required, opened a new log
I20260810 00:50:38.875355 28977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:38.872787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:36011 (local address 127.0.0.1:42083)
0810 00:50:38.872994 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:38.872999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:38.873308 (+ 309us) server_negotiation.cc:407] Connection header received
0810 00:50:38.873464 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:38.873468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:38.873512 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:38.873600 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:38.874604 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:38.874608 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:38.874611 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:38.874626 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:38.874662 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:38.874676 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:38.874786 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:38.874990 (+ 204us) server_negotiation.cc:299] Negotiation successful
0810 00:50:38.875044 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 00:50:38.877447 28974 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe6fe477f25f4f10a67a84ea0fce4c89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6fe477f25f4f10a67a84ea0fce4c89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42083 } }
I20260810 00:50:38.877816 28974 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe6fe477f25f4f10a67a84ea0fce4c89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:50:38.877935 28974 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe6fe477f25f4f10a67a84ea0fce4c89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe6fe477f25f4f10a67a84ea0fce4c89, State: Initialized, Role: FOLLOWER
I20260810 00:50:38.878499 28974 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe6fe477f25f4f10a67a84ea0fce4c89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6fe477f25f4f10a67a84ea0fce4c89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42083 } }
I20260810 00:50:38.878754 28974 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe6fe477f25f4f10a67a84ea0fce4c89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:50:38.878932 28974 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe6fe477f25f4f10a67a84ea0fce4c89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:50:38.879161 28974 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe6fe477f25f4f10a67a84ea0fce4c89 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:50:38.881373 28974 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe6fe477f25f4f10a67a84ea0fce4c89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6fe477f25f4f10a67a84ea0fce4c89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42083 } }
I20260810 00:50:38.881922 28974 leader_election.cc:302] T 00000000000000000000000000000000 P fe6fe477f25f4f10a67a84ea0fce4c89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe6fe477f25f4f10a67a84ea0fce4c89; no voters:
I20260810 00:50:38.882510 28974 leader_election.cc:288] T 00000000000000000000000000000000 P fe6fe477f25f4f10a67a84ea0fce4c89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:50:38.882678 28987 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe6fe477f25f4f10a67a84ea0fce4c89 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:50:38.883553 28987 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe6fe477f25f4f10a67a84ea0fce4c89 [term 1 LEADER]: Becoming Leader. State: Replica: fe6fe477f25f4f10a67a84ea0fce4c89, State: Running, Role: LEADER
I20260810 00:50:38.884133 28987 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe6fe477f25f4f10a67a84ea0fce4c89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6fe477f25f4f10a67a84ea0fce4c89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42083 } }
I20260810 00:50:38.884769 28974 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe6fe477f25f4f10a67a84ea0fce4c89 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:50:38.887437 28992 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe6fe477f25f4f10a67a84ea0fce4c89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe6fe477f25f4f10a67a84ea0fce4c89. Latest consensus state: current_term: 1 leader_uuid: "fe6fe477f25f4f10a67a84ea0fce4c89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6fe477f25f4f10a67a84ea0fce4c89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42083 } } }
I20260810 00:50:38.887841 28992 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe6fe477f25f4f10a67a84ea0fce4c89 [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:38.888604 28989 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe6fe477f25f4f10a67a84ea0fce4c89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe6fe477f25f4f10a67a84ea0fce4c89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6fe477f25f4f10a67a84ea0fce4c89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42083 } } }
I20260810 00:50:38.889047 28989 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe6fe477f25f4f10a67a84ea0fce4c89 [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:38.889415 29001 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:50:38.894421 29001 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:50:38.903331 29001 catalog_manager.cc:1347] Generated new cluster ID: 575ae1f42f5a4e67af6220e5186a1a6e
I20260810 00:50:38.903436 29001 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:50:38.918315 29001 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:50:38.918713 29001 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:50:38.930048 29001 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe6fe477f25f4f10a67a84ea0fce4c89: Generated new TSK 0
I20260810 00:50:38.930986 29001 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 00:50:38.937305 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 28876
W20260810 00:50:38.938494 28969 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.98:51973
W20260810 00:50:38.939266 28696 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 28876: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 00:50:38.939424 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 28876
2026-08-10T00:50:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2140 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:50:39 dist-test-slave-1nl6 krb5kdc[29030](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:50:39 dist-test-slave-1nl6 krb5kdc[29030](info): set up 2 sockets
Aug 10 00:50:39 dist-test-slave-1nl6 krb5kdc[29030](info): commencing operation
krb5kdc: starting...
W20260810 00:50:40.271518 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.287s user 0.003s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:50:40 dist-test-slave-1nl6 krb5kdc[29030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:50:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:50:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 00:50:40.563705 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43429
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:50:40.906272 29049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:50:40.906611 29049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:50:40.906790 29049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:50:40.914178 29049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:50:40.914340 29049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:50:40.914443 29049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:50:40.914549 29049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:50:40.914631 29049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:50:40.924456 29049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43429
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:50:40.925719 29049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:50:40.927032 29049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:50:40.939062 29056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:40.939019 29060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:40.939100 29058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:50:40.940255 29049 server_base.cc:1034] running on GCE node
I20260810 00:50:40.941018 29049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:50:40.942370 29049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:50:40.943608 29049 hybrid_clock.cc:648] HybridClock initialized: now 1786323040943554 us; error 62 us; skew 500 ppm
I20260810 00:50:40.943948 29049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:50:40 dist-test-slave-1nl6 krb5kdc[29030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323040, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:50:40.947113 29049 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 00:50:40.948529 29049 webserver.cc:460] Webserver started at http://127.0.0.1:39729/ using document root <none> and password file <none>
I20260810 00:50:40.949208 29049 fs_manager.cc:359] Metadata directory not provided
I20260810 00:50:40.949373 29049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:50:40.949680 29049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:50:40.952769 29049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "99a63d39807d4e938e848e1514ec6613"
format_stamp: "Formatted at 2026-08-10 00:50:40 on dist-test-slave-1nl6"
server_key: "12ece01e3e7239cc33135951a82a459f"
server_key_iv: "19360c7bd058b5c67e27893315e3f89d"
server_key_version: "encryptionkey@0"
I20260810 00:50:40.953588 29049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "99a63d39807d4e938e848e1514ec6613"
format_stamp: "Formatted at 2026-08-10 00:50:40 on dist-test-slave-1nl6"
server_key: "12ece01e3e7239cc33135951a82a459f"
server_key_iv: "19360c7bd058b5c67e27893315e3f89d"
server_key_version: "encryptionkey@0"
I20260810 00:50:40.958655 29049 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260810 00:50:40.961987 29071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:40.963304 29049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 00:50:40.963512 29049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "99a63d39807d4e938e848e1514ec6613"
format_stamp: "Formatted at 2026-08-10 00:50:40 on dist-test-slave-1nl6"
server_key: "12ece01e3e7239cc33135951a82a459f"
server_key_iv: "19360c7bd058b5c67e27893315e3f89d"
server_key_version: "encryptionkey@0"
I20260810 00:50:40.963713 29049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:50:40.992605 29049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:50:40.996522 29049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:50:40.996807 29049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:50:41.015332 29049 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43429
I20260810 00:50:41.015373 29151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43429 every 8 connection(s)
I20260810 00:50:41.016731 29049 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:50:41.020394 29152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:50:41.024592 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 29049
I20260810 00:50:41.024786 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:50:41.025229 28696 external_mini_cluster.cc:1428] Setting key 38c6ca34145813e61939737b82006fb5
I20260810 00:50:41.031368 29152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99a63d39807d4e938e848e1514ec6613: Bootstrap starting.
I20260810 00:50:41.035030 29152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99a63d39807d4e938e848e1514ec6613: Neither blocks nor log segments found. Creating new log.
I20260810 00:50:41.036201 29152 log.cc:824] T 00000000000000000000000000000000 P 99a63d39807d4e938e848e1514ec6613: Log is configured to *not* fsync() on all Append() calls
I20260810 00:50:41.039103 29152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99a63d39807d4e938e848e1514ec6613: No bootstrap required, opened a new log
Aug 10 00:50:41 dist-test-slave-1nl6 krb5kdc[29030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 00:50:41.047153 29152 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99a63d39807d4e938e848e1514ec6613 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a63d39807d4e938e848e1514ec6613" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43429 } }
I20260810 00:50:41.047662 29152 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99a63d39807d4e938e848e1514ec6613 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:50:41.047788 29152 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99a63d39807d4e938e848e1514ec6613 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a63d39807d4e938e848e1514ec6613, State: Initialized, Role: FOLLOWER
I20260810 00:50:41.048344 29152 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99a63d39807d4e938e848e1514ec6613 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a63d39807d4e938e848e1514ec6613" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43429 } }
I20260810 00:50:41.048604 29152 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99a63d39807d4e938e848e1514ec6613 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:50:41.048789 29152 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99a63d39807d4e938e848e1514ec6613 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:50:41.048811 29156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:41.027129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:49335 (local address 127.0.0.1:43429)
0810 00:50:41.027927 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:41.027939 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:41.027981 (+ 42us) server_negotiation.cc:407] Connection header received
0810 00:50:41.029825 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:41.029856 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:41.030228 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:41.030523 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:41.032435 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:41.033743 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:41.035619 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:41.036271 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:41.036492 (+ 221us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:41.040744 (+ 4252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:41.040752 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:41.040766 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:50:41.040811 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:41.044069 (+ 3258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:41.044981 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:41.044988 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:41.044994 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:41.045164 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:41.045695 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:41.045699 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:41.045701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:41.046247 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:41.046591 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:41.047178 (+ 587us) server_negotiation.cc:299] Negotiation successful
0810 00:50:41.047480 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":369,"threads_started":1}
I20260810 00:50:41.049000 29152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99a63d39807d4e938e848e1514ec6613 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:50:41.051954 29152 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99a63d39807d4e938e848e1514ec6613 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a63d39807d4e938e848e1514ec6613" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43429 } }
I20260810 00:50:41.052500 29152 leader_election.cc:302] T 00000000000000000000000000000000 P 99a63d39807d4e938e848e1514ec6613 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99a63d39807d4e938e848e1514ec6613; no voters:
I20260810 00:50:41.053076 28696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 00:50:41.053196 29152 leader_election.cc:288] T 00000000000000000000000000000000 P 99a63d39807d4e938e848e1514ec6613 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:50:41.053313 29159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99a63d39807d4e938e848e1514ec6613 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:50:41.053977 29159 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99a63d39807d4e938e848e1514ec6613 [term 1 LEADER]: Becoming Leader. State: Replica: 99a63d39807d4e938e848e1514ec6613, State: Running, Role: LEADER
I20260810 00:50:41.054579 29159 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99a63d39807d4e938e848e1514ec6613 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a63d39807d4e938e848e1514ec6613" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43429 } }
I20260810 00:50:41.057005 29152 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99a63d39807d4e938e848e1514ec6613 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:50:41.057331 29163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99a63d39807d4e938e848e1514ec6613 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99a63d39807d4e938e848e1514ec6613" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a63d39807d4e938e848e1514ec6613" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43429 } } }
I20260810 00:50:41.057741 29163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99a63d39807d4e938e848e1514ec6613 [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:41.058266 29162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99a63d39807d4e938e848e1514ec6613 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99a63d39807d4e938e848e1514ec6613. Latest consensus state: current_term: 1 leader_uuid: "99a63d39807d4e938e848e1514ec6613" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a63d39807d4e938e848e1514ec6613" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43429 } } }
I20260810 00:50:41.058615 29162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99a63d39807d4e938e848e1514ec6613 [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:41.069311 29166 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:50:41.075098 29166 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:50:41.095117 29166 catalog_manager.cc:1347] Generated new cluster ID: 4b7b2a3537d24393a4ff4fd47dc888cf
I20260810 00:50:41.095243 29166 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:50:41.100348 29156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:41.074005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:47127 (local address 127.0.0.1:43429)
0810 00:50:41.074190 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:41.074199 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:41.081338 (+ 7139us) server_negotiation.cc:407] Connection header received
0810 00:50:41.081633 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:41.081642 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:41.081743 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:41.081834 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:41.084231 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:41.085809 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:41.088459 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:41.089411 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:41.089538 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:41.092567 (+ 3029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:41.092576 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:41.092582 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:50:41.092619 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:41.096497 (+ 3878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:41.097455 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:41.097461 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:41.097464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:41.097630 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:41.098144 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:41.098149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:41.098152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:41.098482 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:41.098708 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:41.099859 (+ 1151us) server_negotiation.cc:299] Negotiation successful
0810 00:50:41.100028 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 00:50:41.114169 29166 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:50:41.115926 29166 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:50:41.129550 29166 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99a63d39807d4e938e848e1514ec6613: Generated new TSK 0
I20260810 00:50:41.130573 29166 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 00:50:41.137650 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 29049
I20260810 00:50:41.138821 29145 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:50:41.227615 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 29049
W20260810 00:50:41.253614 29024 connection.cc:535] client connection to 127.0.0.1:43429 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43429): Connection reset by peer (error 104)
2026-08-10T00:50:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2287 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:50:41 dist-test-slave-1nl6 krb5kdc[29208](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:50:41 dist-test-slave-1nl6 krb5kdc[29208](info): set up 2 sockets
Aug 10 00:50:41 dist-test-slave-1nl6 krb5kdc[29208](info): commencing operation
krb5kdc: starting...
W20260810 00:50:42.557562 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.285s user 0.000s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:50:42 dist-test-slave-1nl6 krb5kdc[29208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:50:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:50:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 00:50:42.849836 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43735
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:50:43.183139 29226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:50:43.183418 29226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:50:43.183511 29226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:50:43.190372 29226 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:50:43.190480 29226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:50:43.190534 29226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:50:43.190598 29226 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:50:43.190642 29226 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:50:43.199957 29226 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43735
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:50:43.201073 29226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:50:43.202297 29226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:50:43.213713 29233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:43.213742 29235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:43.213900 29232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:50:43.214850 29226 server_base.cc:1034] running on GCE node
I20260810 00:50:43.215627 29226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:50:43.216900 29226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:50:43.218097 29226 hybrid_clock.cc:648] HybridClock initialized: now 1786323043218044 us; error 51 us; skew 500 ppm
I20260810 00:50:43.218456 29226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:50:43 dist-test-slave-1nl6 krb5kdc[29208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323043, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:50:43.224788 29226 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 00:50:43.226217 29226 webserver.cc:460] Webserver started at http://127.0.0.1:41343/ using document root <none> and password file <none>
I20260810 00:50:43.226876 29226 fs_manager.cc:359] Metadata directory not provided
I20260810 00:50:43.227083 29226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:50:43.227396 29226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:50:43.230309 29226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "28263b8726f6443aa2af255ba2326eb5"
format_stamp: "Formatted at 2026-08-10 00:50:43 on dist-test-slave-1nl6"
server_key: "06c64ae8f3cccf2d1bdb910d2607470c"
server_key_iv: "9aa6a977c19ff657d0700db1687b17ac"
server_key_version: "encryptionkey@0"
I20260810 00:50:43.231158 29226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "28263b8726f6443aa2af255ba2326eb5"
format_stamp: "Formatted at 2026-08-10 00:50:43 on dist-test-slave-1nl6"
server_key: "06c64ae8f3cccf2d1bdb910d2607470c"
server_key_iv: "9aa6a977c19ff657d0700db1687b17ac"
server_key_version: "encryptionkey@0"
I20260810 00:50:43.236116 29226 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260810 00:50:43.239364 29245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:43.240617 29226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 00:50:43.240808 29226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "28263b8726f6443aa2af255ba2326eb5"
format_stamp: "Formatted at 2026-08-10 00:50:43 on dist-test-slave-1nl6"
server_key: "06c64ae8f3cccf2d1bdb910d2607470c"
server_key_iv: "9aa6a977c19ff657d0700db1687b17ac"
server_key_version: "encryptionkey@0"
I20260810 00:50:43.241009 29226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:50:43.267258 29226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:50:43.271160 29226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:50:43.271464 29226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:50:43.290544 29226 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43735
I20260810 00:50:43.290582 29326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43735 every 8 connection(s)
I20260810 00:50:43.291960 29226 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:50:43.295337 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 29226
I20260810 00:50:43.295526 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:50:43.295776 29327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:50:43.296109 28696 external_mini_cluster.cc:1428] Setting key 2cec60c2d9e6e50731f1bb270c2d6d26
I20260810 00:50:43.309551 29327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28263b8726f6443aa2af255ba2326eb5: Bootstrap starting.
I20260810 00:50:43.309584 29332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:43.297866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:36295 (local address 127.0.0.1:43735)
0810 00:50:43.298684 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:43.298698 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:43.298745 (+ 47us) server_negotiation.cc:407] Connection header received
0810 00:50:43.300528 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:43.300549 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:43.300935 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:43.301249 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:43.301868 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:43.303262 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:43.305104 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:43.305778 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:43.305933 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:43.307308 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:43.307318 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:43.307332 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:43.307880 (+ 548us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:43.307956 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:43.307979 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:43.308099 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:43.308447 (+ 348us) server_negotiation.cc:299] Negotiation successful
0810 00:50:43.308633 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":355,"threads_started":1}
I20260810 00:50:43.312775 29327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28263b8726f6443aa2af255ba2326eb5: Neither blocks nor log segments found. Creating new log.
I20260810 00:50:43.313400 28696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 00:50:43.313978 29327 log.cc:824] T 00000000000000000000000000000000 P 28263b8726f6443aa2af255ba2326eb5: Log is configured to *not* fsync() on all Append() calls
I20260810 00:50:43.317294 29327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28263b8726f6443aa2af255ba2326eb5: No bootstrap required, opened a new log
I20260810 00:50:43.325332 29332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:43.318714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:45315 (local address 127.0.0.1:43735)
0810 00:50:43.318888 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:43.318893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:43.319327 (+ 434us) server_negotiation.cc:407] Connection header received
0810 00:50:43.319470 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:43.319475 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:43.319553 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:43.319669 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:43.320132 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:43.321300 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:43.322600 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:43.323340 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:43.323470 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:43.324566 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:43.324573 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:43.324578 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:43.324599 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:43.324645 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:43.324665 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:43.324815 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:43.324965 (+ 150us) server_negotiation.cc:299] Negotiation successful
0810 00:50:43.325017 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 00:50:43.327211 29327 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28263b8726f6443aa2af255ba2326eb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28263b8726f6443aa2af255ba2326eb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43735 } }
I20260810 00:50:43.327528 29327 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28263b8726f6443aa2af255ba2326eb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:50:43.327606 29327 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28263b8726f6443aa2af255ba2326eb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28263b8726f6443aa2af255ba2326eb5, State: Initialized, Role: FOLLOWER
I20260810 00:50:43.328063 29327 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28263b8726f6443aa2af255ba2326eb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28263b8726f6443aa2af255ba2326eb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43735 } }
I20260810 00:50:43.328284 29327 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28263b8726f6443aa2af255ba2326eb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:50:43.328384 29327 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28263b8726f6443aa2af255ba2326eb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:50:43.328501 29327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28263b8726f6443aa2af255ba2326eb5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:50:43.330755 29327 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28263b8726f6443aa2af255ba2326eb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28263b8726f6443aa2af255ba2326eb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43735 } }
I20260810 00:50:43.331389 29327 leader_election.cc:302] T 00000000000000000000000000000000 P 28263b8726f6443aa2af255ba2326eb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28263b8726f6443aa2af255ba2326eb5; no voters:
I20260810 00:50:43.331882 29327 leader_election.cc:288] T 00000000000000000000000000000000 P 28263b8726f6443aa2af255ba2326eb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:50:43.332023 29341 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28263b8726f6443aa2af255ba2326eb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:50:43.332837 29341 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28263b8726f6443aa2af255ba2326eb5 [term 1 LEADER]: Becoming Leader. State: Replica: 28263b8726f6443aa2af255ba2326eb5, State: Running, Role: LEADER
I20260810 00:50:43.333361 29341 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28263b8726f6443aa2af255ba2326eb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28263b8726f6443aa2af255ba2326eb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43735 } }
I20260810 00:50:43.334167 29327 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28263b8726f6443aa2af255ba2326eb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:50:43.336619 29342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28263b8726f6443aa2af255ba2326eb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28263b8726f6443aa2af255ba2326eb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28263b8726f6443aa2af255ba2326eb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43735 } } }
I20260810 00:50:43.336606 29344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28263b8726f6443aa2af255ba2326eb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28263b8726f6443aa2af255ba2326eb5. Latest consensus state: current_term: 1 leader_uuid: "28263b8726f6443aa2af255ba2326eb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28263b8726f6443aa2af255ba2326eb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43735 } } }
I20260810 00:50:43.337126 29344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28263b8726f6443aa2af255ba2326eb5 [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:43.337124 29342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28263b8726f6443aa2af255ba2326eb5 [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:43.339752 29349 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:50:43.344987 29349 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:50:43.354043 29349 catalog_manager.cc:1347] Generated new cluster ID: ffbccd4f2d5d42b5a52ba4dc757ed2ce
I20260810 00:50:43.354158 29349 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:50:43.373787 29349 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:50:43.375352 29349 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:50:43.382503 29349 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28263b8726f6443aa2af255ba2326eb5: Generated new TSK 0
I20260810 00:50:43.383296 29349 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 00:50:43.389952 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 29226
W20260810 00:50:43.391175 29321 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.98:36295
W20260810 00:50:43.391996 28696 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 29226: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 00:50:43.392172 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 29226
W20260810 00:50:43.414353 29204 connection.cc:535] client connection to 127.0.0.1:43735 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43735): Connection reset by peer (error 104)
2026-08-10T00:50:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2160 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:50:43 dist-test-slave-1nl6 krb5kdc[29377](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:50:43 dist-test-slave-1nl6 krb5kdc[29377](info): set up 2 sockets
Aug 10 00:50:43 dist-test-slave-1nl6 krb5kdc[29377](info): commencing operation
krb5kdc: starting...
W20260810 00:50:44.695983 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.263s user 0.000s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:50:44 dist-test-slave-1nl6 krb5kdc[29377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:50:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:50:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 00:50:44.975955 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36627
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:50:45.309284 29402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:50:45.309543 29402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:50:45.309631 29402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:50:45.316548 29402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:50:45.316644 29402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:50:45.316697 29402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:50:45.316751 29402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:50:45.316795 29402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:50:45.326061 29402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36627
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786322687226232-28696-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:50:45.327224 29402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:50:45.328399 29402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:50:45.339926 29415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:45.340029 29412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:45.340000 29411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:50:45.340754 29402 server_base.cc:1034] running on GCE node
I20260810 00:50:45.341467 29402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:50:45.342718 29402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:50:45.343916 29402 hybrid_clock.cc:648] HybridClock initialized: now 1786323045343852 us; error 60 us; skew 500 ppm
I20260810 00:50:45.344277 29402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:50:45 dist-test-slave-1nl6 krb5kdc[29377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:50:45.347467 29402 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 00:50:45.348868 29402 webserver.cc:460] Webserver started at http://127.0.0.1:37581/ using document root <none> and password file <none>
I20260810 00:50:45.349519 29402 fs_manager.cc:359] Metadata directory not provided
I20260810 00:50:45.349671 29402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:50:45.349977 29402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:50:45.352921 29402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "9d874f17cf894b51aadcfe050de617a4"
format_stamp: "Formatted at 2026-08-10 00:50:45 on dist-test-slave-1nl6"
server_key: "a2f16ddd04f89e477292d41ff1d60bc3"
server_key_iv: "ddf31d32d071ce017f413772a3a040f1"
server_key_version: "encryptionkey@0"
I20260810 00:50:45.353686 29402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "9d874f17cf894b51aadcfe050de617a4"
format_stamp: "Formatted at 2026-08-10 00:50:45 on dist-test-slave-1nl6"
server_key: "a2f16ddd04f89e477292d41ff1d60bc3"
server_key_iv: "ddf31d32d071ce017f413772a3a040f1"
server_key_version: "encryptionkey@0"
I20260810 00:50:45.358651 29402 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260810 00:50:45.361956 29427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:45.363269 29402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 00:50:45.363445 29402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "9d874f17cf894b51aadcfe050de617a4"
format_stamp: "Formatted at 2026-08-10 00:50:45 on dist-test-slave-1nl6"
server_key: "a2f16ddd04f89e477292d41ff1d60bc3"
server_key_iv: "ddf31d32d071ce017f413772a3a040f1"
server_key_version: "encryptionkey@0"
I20260810 00:50:45.363646 29402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:50:45.369027 29402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:50:45.373126 29402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:50:45.373420 29402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:50:45.391289 29402 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36627
I20260810 00:50:45.391361 29504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36627 every 8 connection(s)
I20260810 00:50:45.392607 29402 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:50:45.396145 29506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:50:45.402546 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 29402
I20260810 00:50:45.402733 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:50:45.403273 28696 external_mini_cluster.cc:1428] Setting key 88db47f72ed2b46d58b8fe35dbfc21e9
I20260810 00:50:45.406469 29506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d874f17cf894b51aadcfe050de617a4: Bootstrap starting.
I20260810 00:50:45.409971 29506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d874f17cf894b51aadcfe050de617a4: Neither blocks nor log segments found. Creating new log.
I20260810 00:50:45.410660 29510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:45.404910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:49735 (local address 127.0.0.1:36627)
0810 00:50:45.405698 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:45.405709 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:45.405756 (+ 47us) server_negotiation.cc:407] Connection header received
0810 00:50:45.407449 (+ 1693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:45.407470 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:45.407779 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:45.408091 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:45.409133 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:45.409139 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:45.409151 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:45.409165 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:45.409214 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:45.409233 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:45.409367 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:45.409546 (+ 179us) server_negotiation.cc:299] Negotiation successful
0810 00:50:45.409765 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":369,"threads_started":1}
I20260810 00:50:45.411175 29506 log.cc:824] T 00000000000000000000000000000000 P 9d874f17cf894b51aadcfe050de617a4: Log is configured to *not* fsync() on all Append() calls
I20260810 00:50:45.414000 29506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d874f17cf894b51aadcfe050de617a4: No bootstrap required, opened a new log
I20260810 00:50:45.414366 28696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 00:50:45.421272 29506 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d874f17cf894b51aadcfe050de617a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d874f17cf894b51aadcfe050de617a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36627 } }
I20260810 00:50:45.421754 29506 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d874f17cf894b51aadcfe050de617a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:50:45.421875 29506 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d874f17cf894b51aadcfe050de617a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d874f17cf894b51aadcfe050de617a4, State: Initialized, Role: FOLLOWER
I20260810 00:50:45.422500 29506 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d874f17cf894b51aadcfe050de617a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d874f17cf894b51aadcfe050de617a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36627 } }
I20260810 00:50:45.422787 29506 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d874f17cf894b51aadcfe050de617a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:50:45.422960 29506 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d874f17cf894b51aadcfe050de617a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:50:45.423159 29506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d874f17cf894b51aadcfe050de617a4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:50:45.423995 29510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:45.421424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:37309 (local address 127.0.0.1:36627)
0810 00:50:45.421603 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:45.421607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:45.421891 (+ 284us) server_negotiation.cc:407] Connection header received
0810 00:50:45.422115 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:45.422118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:45.422161 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:45.422277 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:45.423263 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:45.423269 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:45.423273 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:45.423292 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:45.423333 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:45.423352 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:45.423509 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:45.423602 (+ 93us) server_negotiation.cc:299] Negotiation successful
0810 00:50:45.423677 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 00:50:45.425518 29506 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d874f17cf894b51aadcfe050de617a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d874f17cf894b51aadcfe050de617a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36627 } }
I20260810 00:50:45.426007 29506 leader_election.cc:302] T 00000000000000000000000000000000 P 9d874f17cf894b51aadcfe050de617a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d874f17cf894b51aadcfe050de617a4; no voters:
I20260810 00:50:45.426594 29506 leader_election.cc:288] T 00000000000000000000000000000000 P 9d874f17cf894b51aadcfe050de617a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:50:45.426729 29518 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d874f17cf894b51aadcfe050de617a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:50:45.427404 29518 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d874f17cf894b51aadcfe050de617a4 [term 1 LEADER]: Becoming Leader. State: Replica: 9d874f17cf894b51aadcfe050de617a4, State: Running, Role: LEADER
I20260810 00:50:45.428020 29518 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d874f17cf894b51aadcfe050de617a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d874f17cf894b51aadcfe050de617a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36627 } }
I20260810 00:50:45.429116 29506 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d874f17cf894b51aadcfe050de617a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:50:45.430927 29523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d874f17cf894b51aadcfe050de617a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d874f17cf894b51aadcfe050de617a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d874f17cf894b51aadcfe050de617a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36627 } } }
I20260810 00:50:45.431372 29523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d874f17cf894b51aadcfe050de617a4 [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:45.430888 29519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d874f17cf894b51aadcfe050de617a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d874f17cf894b51aadcfe050de617a4. Latest consensus state: current_term: 1 leader_uuid: "9d874f17cf894b51aadcfe050de617a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d874f17cf894b51aadcfe050de617a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36627 } } }
I20260810 00:50:45.431834 29519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d874f17cf894b51aadcfe050de617a4 [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:45.432686 29527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:50:45.437462 29527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:50:45.445513 29527 catalog_manager.cc:1347] Generated new cluster ID: bfcbb0bf5b4740b8a3a9eede1d341197
I20260810 00:50:45.445612 29527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:50:45.475339 29527 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:50:45.475751 29527 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:50:45.483527 29527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d874f17cf894b51aadcfe050de617a4: Generated new TSK 0
I20260810 00:50:45.484241 29527 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 00:50:45.495858 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 29402
W20260810 00:50:45.496898 29498 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.98:49735
W20260810 00:50:45.497575 28696 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 29402: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 00:50:45.497725 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 29402
2026-08-10T00:50:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2109 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:50:45 dist-test-slave-1nl6 krb5kdc[29551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:50:45 dist-test-slave-1nl6 krb5kdc[29551](info): set up 2 sockets
Aug 10 00:50:45 dist-test-slave-1nl6 krb5kdc[29551](info): commencing operation
krb5kdc: starting...
W20260810 00:50:46.806177 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.263s user 0.007s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:50:47 dist-test-slave-1nl6 krb5kdc[29551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:50:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:50:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.6.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
I20260810 00:50:47.142326 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:43491
--webserver_interface=127.28.6.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:33327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:43491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:50:47.492775 29572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:50:47.493047 29572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:50:47.493201 29572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:50:47.500224 29572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:50:47.500332 29572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:50:47.500429 29572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:50:47.500526 29572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:50:47.500602 29572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:50:47.510341 29572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:33327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:43491
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:43491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:50:47.511627 29572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:50:47.512940 29572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:50:47.525082 29578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:47.525111 29579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:47.525301 29581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:50:47.526057 29572 server_base.cc:1034] running on GCE node
I20260810 00:50:47.526844 29572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:50:47.528231 29572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:50:47.529475 29572 hybrid_clock.cc:648] HybridClock initialized: now 1786323047529420 us; error 49 us; skew 500 ppm
I20260810 00:50:47.529923 29572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:50:47 dist-test-slave-1nl6 krb5kdc[29551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323047, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:50:47.533355 29572 init.cc:377] Logged in from keytab as kudu/127.28.6.62@KRBTEST.COM (short username kudu)
I20260810 00:50:47.535228 29572 webserver.cc:460] Webserver started at http://127.28.6.62:44053/ using document root <none> and password file <none>
I20260810 00:50:47.536118 29572 fs_manager.cc:359] Metadata directory not provided
I20260810 00:50:47.536283 29572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:50:47.536716 29572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:50:47.540866 29572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "41689fb804f440f5911a6daff3519f03"
format_stamp: "Formatted at 2026-08-10 00:50:47 on dist-test-slave-1nl6"
server_key: "5a6f93b18bb46b02dff0385bad09fff3"
server_key_iv: "0a064f2263e3e606049d900760536f55"
server_key_version: "encryptionkey@0"
I20260810 00:50:47.541673 29572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "41689fb804f440f5911a6daff3519f03"
format_stamp: "Formatted at 2026-08-10 00:50:47 on dist-test-slave-1nl6"
server_key: "5a6f93b18bb46b02dff0385bad09fff3"
server_key_iv: "0a064f2263e3e606049d900760536f55"
server_key_version: "encryptionkey@0"
I20260810 00:50:47.546591 29572 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260810 00:50:47.549883 29595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:47.551363 29572 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 00:50:47.551517 29572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "41689fb804f440f5911a6daff3519f03"
format_stamp: "Formatted at 2026-08-10 00:50:47 on dist-test-slave-1nl6"
server_key: "5a6f93b18bb46b02dff0385bad09fff3"
server_key_iv: "0a064f2263e3e606049d900760536f55"
server_key_version: "encryptionkey@0"
I20260810 00:50:47.551672 29572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:50:47.571156 29572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:50:47.575296 29572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:50:47.575536 29572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:50:47.593271 29572 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:43491
I20260810 00:50:47.593312 29675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:43491 every 8 connection(s)
I20260810 00:50:47.594571 29572 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:50:47.594877 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 29572
I20260810 00:50:47.595093 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:50:47.595723 28696 external_mini_cluster.cc:1428] Setting key 7045b99ba19e4128f5da12718723d5d9
I20260810 00:50:47.598984 29676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 00:50:47 dist-test-slave-1nl6 krb5kdc[29551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:50:47.609051 29676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41689fb804f440f5911a6daff3519f03: Bootstrap starting.
I20260810 00:50:47.612077 29676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41689fb804f440f5911a6daff3519f03: Neither blocks nor log segments found. Creating new log.
I20260810 00:50:47.613240 29676 log.cc:824] T 00000000000000000000000000000000 P 41689fb804f440f5911a6daff3519f03: Log is configured to *not* fsync() on all Append() calls
I20260810 00:50:47.616727 29676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41689fb804f440f5911a6daff3519f03: No bootstrap required, opened a new log
I20260810 00:50:47.617785 29680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:47.597540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55087 (local address 127.28.6.62:43491)
0810 00:50:47.598327 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:47.598338 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:47.598379 (+ 41us) server_negotiation.cc:407] Connection header received
0810 00:50:47.600088 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:47.600110 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:47.600480 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:47.600858 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:47.602298 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:47.603735 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:47.605012 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:47.605652 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:47.605863 (+ 211us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:47.610105 (+ 4242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:47.610112 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:47.610125 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:50:47.610159 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:47.613318 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:47.614518 (+ 1200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:47.614528 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:47.614537 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:47.614755 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:47.615313 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:47.615317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:47.615319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:47.615753 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:47.615993 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:47.616508 (+ 515us) server_negotiation.cc:299] Negotiation successful
0810 00:50:47.616783 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":361,"threads_started":1}
I20260810 00:50:47.621833 28696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 00:50:47.626828 29676 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41689fb804f440f5911a6daff3519f03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41689fb804f440f5911a6daff3519f03" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 43491 } }
I20260810 00:50:47.627267 29676 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41689fb804f440f5911a6daff3519f03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:50:47.627380 29676 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41689fb804f440f5911a6daff3519f03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41689fb804f440f5911a6daff3519f03, State: Initialized, Role: FOLLOWER
I20260810 00:50:47.628051 29676 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41689fb804f440f5911a6daff3519f03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41689fb804f440f5911a6daff3519f03" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 43491 } }
I20260810 00:50:47.628371 29676 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41689fb804f440f5911a6daff3519f03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:50:47.628546 29676 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41689fb804f440f5911a6daff3519f03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:50:47.628726 29676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41689fb804f440f5911a6daff3519f03 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:50:47.630828 29676 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41689fb804f440f5911a6daff3519f03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41689fb804f440f5911a6daff3519f03" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 43491 } }
I20260810 00:50:47.631373 29676 leader_election.cc:302] T 00000000000000000000000000000000 P 41689fb804f440f5911a6daff3519f03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41689fb804f440f5911a6daff3519f03; no voters:
I20260810 00:50:47.633142 29676 leader_election.cc:288] T 00000000000000000000000000000000 P 41689fb804f440f5911a6daff3519f03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:50:47.633217 29692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41689fb804f440f5911a6daff3519f03 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:50:47.636726 29676 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41689fb804f440f5911a6daff3519f03 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:50:47.638279 29692 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41689fb804f440f5911a6daff3519f03 [term 1 LEADER]: Becoming Leader. State: Replica: 41689fb804f440f5911a6daff3519f03, State: Running, Role: LEADER
I20260810 00:50:47.638959 29692 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41689fb804f440f5911a6daff3519f03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41689fb804f440f5911a6daff3519f03" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 43491 } }
I20260810 00:50:47.646667 29680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:47.627144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42689 (local address 127.28.6.62:43491)
0810 00:50:47.627299 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:47.627303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:47.628015 (+ 712us) server_negotiation.cc:407] Connection header received
0810 00:50:47.628188 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:47.628192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:47.628257 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:47.628378 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:47.629989 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:47.630831 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:47.632803 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:47.634387 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:47.634538 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:47.638080 (+ 3542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:47.638089 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:47.638096 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:50:47.638148 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:47.642485 (+ 4337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:47.643737 (+ 1252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:47.643745 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:47.643752 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:47.644019 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:47.644666 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:47.644673 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:47.644679 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:47.645145 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:47.645540 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:47.645995 (+ 455us) server_negotiation.cc:299] Negotiation successful
0810 00:50:47.646216 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 00:50:47.650029 29694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41689fb804f440f5911a6daff3519f03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41689fb804f440f5911a6daff3519f03. Latest consensus state: current_term: 1 leader_uuid: "41689fb804f440f5911a6daff3519f03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41689fb804f440f5911a6daff3519f03" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 43491 } } }
I20260810 00:50:47.650429 29694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41689fb804f440f5911a6daff3519f03 [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:47.651301 29708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:50:47.651139 29693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41689fb804f440f5911a6daff3519f03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41689fb804f440f5911a6daff3519f03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41689fb804f440f5911a6daff3519f03" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 43491 } } }
I20260810 00:50:47.651579 29693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41689fb804f440f5911a6daff3519f03 [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:47.656158 29708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:50:47.664268 29708 catalog_manager.cc:1347] Generated new cluster ID: 9a33b0a681c44dc68689c5c602d9e426
I20260810 00:50:47.664373 29708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:50:47.684100 29708 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:50:47.686050 29708 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:50:47.702481 29708 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41689fb804f440f5911a6daff3519f03: Generated new TSK 0
I20260810 00:50:47.703671 29708 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 00:50:47.720605 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 29572
I20260810 00:50:47.721865 29667 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:50:47.815876 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 29572
W20260810 00:50:47.842190 29549 connection.cc:535] client connection to 127.28.6.62:43491 recv error: Network error: failed to read from TLS socket (remote: 127.28.6.62:43491): Connection reset by peer (error 104)
2026-08-10T00:50:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2317 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:50:47 dist-test-slave-1nl6 krb5kdc[29721](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:50:47 dist-test-slave-1nl6 krb5kdc[29721](info): set up 2 sockets
Aug 10 00:50:47 dist-test-slave-1nl6 krb5kdc[29721](info): commencing operation
krb5kdc: starting...
W20260810 00:50:49.147323 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.286s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:50:49 dist-test-slave-1nl6 krb5kdc[29721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:50:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:50:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.6.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
I20260810 00:50:49.448042 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:38439
--webserver_interface=127.28.6.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:42991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:38439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:50:49.795979 29738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:50:49.796288 29738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:50:49.796456 29738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:50:49.803659 29738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:50:49.803773 29738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:50:49.803853 29738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:50:49.803953 29738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:50:49.804026 29738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:50:49.814062 29738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:42991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:38439
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:38439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:50:49.815492 29738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:50:49.816824 29738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:50:49.828469 29750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:49.828634 29745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:49.828567 29746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:50:49.829670 29738 server_base.cc:1034] running on GCE node
I20260810 00:50:49.830436 29738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:50:49.831856 29738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:50:49.833086 29738 hybrid_clock.cc:648] HybridClock initialized: now 1786323049833051 us; error 46 us; skew 500 ppm
I20260810 00:50:49.833431 29738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:50:49 dist-test-slave-1nl6 krb5kdc[29721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323049, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:50:49.842185 29738 init.cc:377] Logged in from keytab as kudu/127.28.6.62@KRBTEST.COM (short username kudu)
I20260810 00:50:49.843621 29738 webserver.cc:460] Webserver started at http://127.28.6.62:33717/ using document root <none> and password file <none>
I20260810 00:50:49.844255 29738 fs_manager.cc:359] Metadata directory not provided
I20260810 00:50:49.844362 29738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:50:49.844614 29738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:50:49.847563 29738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "4c2afa765ccf46d9a0067350817a3832"
format_stamp: "Formatted at 2026-08-10 00:50:49 on dist-test-slave-1nl6"
server_key: "2f43cef575b806f7886b4f0da12620b6"
server_key_iv: "a63dc3e362b3a3ef8db0fdd6d8a311c9"
server_key_version: "encryptionkey@0"
I20260810 00:50:49.848284 29738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "4c2afa765ccf46d9a0067350817a3832"
format_stamp: "Formatted at 2026-08-10 00:50:49 on dist-test-slave-1nl6"
server_key: "2f43cef575b806f7886b4f0da12620b6"
server_key_iv: "a63dc3e362b3a3ef8db0fdd6d8a311c9"
server_key_version: "encryptionkey@0"
I20260810 00:50:49.853251 29738 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 00:50:49.856554 29759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:49.857839 29738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 00:50:49.858017 29738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "4c2afa765ccf46d9a0067350817a3832"
format_stamp: "Formatted at 2026-08-10 00:50:49 on dist-test-slave-1nl6"
server_key: "2f43cef575b806f7886b4f0da12620b6"
server_key_iv: "a63dc3e362b3a3ef8db0fdd6d8a311c9"
server_key_version: "encryptionkey@0"
I20260810 00:50:49.858198 29738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:50:49.886816 29738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:50:49.891105 29738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:50:49.891395 29738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:50:49.910620 29738 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:38439
I20260810 00:50:49.910704 29845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:38439 every 8 connection(s)
I20260810 00:50:49.911988 29738 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:50:49.915793 29848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:50:49.918184 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 29738
I20260810 00:50:49.918383 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:50:49.918807 28696 external_mini_cluster.cc:1428] Setting key 0569e4df5f922cdda24165278b0c0a9c
I20260810 00:50:49.927893 29848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c2afa765ccf46d9a0067350817a3832: Bootstrap starting.
I20260810 00:50:49.931694 29848 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c2afa765ccf46d9a0067350817a3832: Neither blocks nor log segments found. Creating new log.
I20260810 00:50:49.932379 29851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:49.920557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42371 (local address 127.28.6.62:38439)
0810 00:50:49.921371 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:49.921383 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:49.921431 (+ 48us) server_negotiation.cc:407] Connection header received
0810 00:50:49.923096 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:49.923118 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:49.923489 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:49.923782 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:49.924437 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:49.925808 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:49.927700 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:49.928396 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:49.928525 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:49.930016 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:49.930023 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:49.930037 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:49.930569 (+ 532us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:49.930648 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:49.930672 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:49.930811 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:49.931130 (+ 319us) server_negotiation.cc:299] Negotiation successful
0810 00:50:49.931316 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":404,"threads_started":1}
I20260810 00:50:49.932937 29848 log.cc:824] T 00000000000000000000000000000000 P 4c2afa765ccf46d9a0067350817a3832: Log is configured to *not* fsync() on all Append() calls
I20260810 00:50:49.936040 29848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c2afa765ccf46d9a0067350817a3832: No bootstrap required, opened a new log
I20260810 00:50:49.936432 28696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 00:50:49.945468 29848 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c2afa765ccf46d9a0067350817a3832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2afa765ccf46d9a0067350817a3832" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 38439 } }
I20260810 00:50:49.945909 29848 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c2afa765ccf46d9a0067350817a3832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:50:49.946023 29848 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c2afa765ccf46d9a0067350817a3832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c2afa765ccf46d9a0067350817a3832, State: Initialized, Role: FOLLOWER
I20260810 00:50:49.946696 29848 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c2afa765ccf46d9a0067350817a3832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2afa765ccf46d9a0067350817a3832" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 38439 } }
I20260810 00:50:49.947021 29848 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c2afa765ccf46d9a0067350817a3832 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:50:49.947237 29848 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c2afa765ccf46d9a0067350817a3832 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:50:49.947558 29848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c2afa765ccf46d9a0067350817a3832 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:50:49.949329 29851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:49.941425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35401 (local address 127.28.6.62:38439)
0810 00:50:49.941616 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:49.941622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:49.942131 (+ 509us) server_negotiation.cc:407] Connection header received
0810 00:50:49.942290 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:49.942296 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:49.942378 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:49.942503 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:49.943224 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:49.944539 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:49.946197 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:49.946705 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:49.946864 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:49.948519 (+ 1655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:49.948527 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:49.948530 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:49.948551 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:49.948614 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:49.948631 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:49.948764 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:49.948981 (+ 217us) server_negotiation.cc:299] Negotiation successful
0810 00:50:49.949042 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 00:50:49.950601 29848 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c2afa765ccf46d9a0067350817a3832 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2afa765ccf46d9a0067350817a3832" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 38439 } }
I20260810 00:50:49.951224 29848 leader_election.cc:302] T 00000000000000000000000000000000 P 4c2afa765ccf46d9a0067350817a3832 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c2afa765ccf46d9a0067350817a3832; no voters:
I20260810 00:50:49.951753 29848 leader_election.cc:288] T 00000000000000000000000000000000 P 4c2afa765ccf46d9a0067350817a3832 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:50:49.951917 29863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c2afa765ccf46d9a0067350817a3832 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:50:49.952764 29863 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c2afa765ccf46d9a0067350817a3832 [term 1 LEADER]: Becoming Leader. State: Replica: 4c2afa765ccf46d9a0067350817a3832, State: Running, Role: LEADER
I20260810 00:50:49.953362 29863 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c2afa765ccf46d9a0067350817a3832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2afa765ccf46d9a0067350817a3832" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 38439 } }
I20260810 00:50:49.953998 29848 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c2afa765ccf46d9a0067350817a3832 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:50:49.956329 29866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c2afa765ccf46d9a0067350817a3832 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c2afa765ccf46d9a0067350817a3832. Latest consensus state: current_term: 1 leader_uuid: "4c2afa765ccf46d9a0067350817a3832" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2afa765ccf46d9a0067350817a3832" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 38439 } } }
I20260810 00:50:49.956930 29866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c2afa765ccf46d9a0067350817a3832 [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:49.958760 29865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c2afa765ccf46d9a0067350817a3832 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c2afa765ccf46d9a0067350817a3832" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2afa765ccf46d9a0067350817a3832" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 38439 } } }
I20260810 00:50:49.959121 29869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:50:49.959152 29865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c2afa765ccf46d9a0067350817a3832 [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:49.964054 29869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:50:49.975052 29869 catalog_manager.cc:1347] Generated new cluster ID: 2bf53be291bf4531bbca561fc681d06b
I20260810 00:50:49.975188 29869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:50:49.994005 29869 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:50:49.995383 29869 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:50:50.004431 29869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c2afa765ccf46d9a0067350817a3832: Generated new TSK 0
I20260810 00:50:50.005373 29869 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 00:50:50.013568 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 29738
W20260810 00:50:50.015211 29835 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42371
W20260810 00:50:50.016259 28696 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 29738: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 00:50:50.016407 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 29738
W20260810 00:50:50.043869 29717 connection.cc:535] client connection to 127.28.6.62:38439 recv error: Network error: failed to read from TLS socket (remote: 127.28.6.62:38439): Connection reset by peer (error 104)
2026-08-10T00:50:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2202 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:50:50 dist-test-slave-1nl6 krb5kdc[29890](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:50:50 dist-test-slave-1nl6 krb5kdc[29890](info): set up 2 sockets
Aug 10 00:50:50 dist-test-slave-1nl6 krb5kdc[29890](info): commencing operation
krb5kdc: starting...
W20260810 00:50:51.365868 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.302s user 0.000s sys 0.053s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:50:51 dist-test-slave-1nl6 krb5kdc[29890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:50:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:50:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.6.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
I20260810 00:50:51.667922 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:37049
--webserver_interface=127.28.6.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:33173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:37049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:50:52.018879 29908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:50:52.019261 29908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:50:52.019433 29908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:50:52.026734 29908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:50:52.026841 29908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:50:52.026978 29908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:50:52.027086 29908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:50:52.027146 29908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:50:52.037251 29908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:33173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:37049
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:37049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:50:52.038540 29908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:50:52.039952 29908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:50:52.052160 29916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:52.052326 29917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:52.052307 29920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:50:52.053099 29908 server_base.cc:1034] running on GCE node
I20260810 00:50:52.053800 29908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:50:52.055142 29908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:50:52.056353 29908 hybrid_clock.cc:648] HybridClock initialized: now 1786323052056315 us; error 24 us; skew 500 ppm
I20260810 00:50:52.056705 29908 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:50:52 dist-test-slave-1nl6 krb5kdc[29890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323052, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:50:52.060142 29908 init.cc:377] Logged in from keytab as kudu/127.28.6.62@KRBTEST.COM (short username kudu)
I20260810 00:50:52.061565 29908 webserver.cc:460] Webserver started at http://127.28.6.62:38275/ using document root <none> and password file <none>
I20260810 00:50:52.062239 29908 fs_manager.cc:359] Metadata directory not provided
I20260810 00:50:52.062386 29908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:50:52.062706 29908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:50:52.065686 29908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "5d52892bb53e4f8dbf4567bd76dadfe3"
format_stamp: "Formatted at 2026-08-10 00:50:52 on dist-test-slave-1nl6"
server_key: "6cdf29b3a534c38bd6761ce9e68126c6"
server_key_iv: "3b746ebe045893cc0b1fcd53eaf98514"
server_key_version: "encryptionkey@0"
I20260810 00:50:52.066447 29908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "5d52892bb53e4f8dbf4567bd76dadfe3"
format_stamp: "Formatted at 2026-08-10 00:50:52 on dist-test-slave-1nl6"
server_key: "6cdf29b3a534c38bd6761ce9e68126c6"
server_key_iv: "3b746ebe045893cc0b1fcd53eaf98514"
server_key_version: "encryptionkey@0"
I20260810 00:50:52.071480 29908 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 00:50:52.074884 29930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:52.076303 29908 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 00:50:52.076478 29908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "5d52892bb53e4f8dbf4567bd76dadfe3"
format_stamp: "Formatted at 2026-08-10 00:50:52 on dist-test-slave-1nl6"
server_key: "6cdf29b3a534c38bd6761ce9e68126c6"
server_key_iv: "3b746ebe045893cc0b1fcd53eaf98514"
server_key_version: "encryptionkey@0"
I20260810 00:50:52.076660 29908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:50:52.082175 29908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:50:52.086318 29908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:50:52.086613 29908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:50:52.103941 29908 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:37049
I20260810 00:50:52.103989 30025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:37049 every 8 connection(s)
I20260810 00:50:52.105305 29908 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:50:52.106232 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 29908
I20260810 00:50:52.106434 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:50:52.106844 28696 external_mini_cluster.cc:1428] Setting key 46f503998f1ee9a1fc5c36c3ccab0cec
I20260810 00:50:52.109582 30030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:50:52.115188 30036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:52.108621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44245 (local address 127.28.6.62:37049)
0810 00:50:52.109414 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:52.109426 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:52.109469 (+ 43us) server_negotiation.cc:407] Connection header received
0810 00:50:52.111726 (+ 2257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:52.111750 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:52.112078 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:52.112533 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:52.113578 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:52.113585 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:52.113597 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:52.113615 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:52.113703 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:52.113724 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:52.113958 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:52.114121 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 00:50:52.114285 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":338,"threads_started":1}
I20260810 00:50:52.119021 28696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 00:50:52.122946 30030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d52892bb53e4f8dbf4567bd76dadfe3: Bootstrap starting.
I20260810 00:50:52.126678 30036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:52.124058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35619 (local address 127.28.6.62:37049)
0810 00:50:52.124211 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:52.124216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:52.124707 (+ 491us) server_negotiation.cc:407] Connection header received
0810 00:50:52.124820 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:52.124827 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:52.124904 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:52.125027 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:52.125935 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:52.125942 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:52.125947 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:52.125966 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:52.126015 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:52.126039 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:52.126171 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:52.126274 (+ 103us) server_negotiation.cc:299] Negotiation successful
0810 00:50:52.126337 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 00:50:52.127341 30030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d52892bb53e4f8dbf4567bd76dadfe3: Neither blocks nor log segments found. Creating new log.
I20260810 00:50:52.128459 30030 log.cc:824] T 00000000000000000000000000000000 P 5d52892bb53e4f8dbf4567bd76dadfe3: Log is configured to *not* fsync() on all Append() calls
I20260810 00:50:52.131256 30030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d52892bb53e4f8dbf4567bd76dadfe3: No bootstrap required, opened a new log
I20260810 00:50:52.140647 30030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d52892bb53e4f8dbf4567bd76dadfe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d52892bb53e4f8dbf4567bd76dadfe3" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 37049 } }
I20260810 00:50:52.141192 30030 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d52892bb53e4f8dbf4567bd76dadfe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:50:52.141312 30030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d52892bb53e4f8dbf4567bd76dadfe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d52892bb53e4f8dbf4567bd76dadfe3, State: Initialized, Role: FOLLOWER
I20260810 00:50:52.141992 30030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d52892bb53e4f8dbf4567bd76dadfe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d52892bb53e4f8dbf4567bd76dadfe3" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 37049 } }
I20260810 00:50:52.142273 30030 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d52892bb53e4f8dbf4567bd76dadfe3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:50:52.142445 30030 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d52892bb53e4f8dbf4567bd76dadfe3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:50:52.142680 30030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d52892bb53e4f8dbf4567bd76dadfe3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:50:52.145638 30030 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d52892bb53e4f8dbf4567bd76dadfe3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d52892bb53e4f8dbf4567bd76dadfe3" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 37049 } }
I20260810 00:50:52.146152 30030 leader_election.cc:302] T 00000000000000000000000000000000 P 5d52892bb53e4f8dbf4567bd76dadfe3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d52892bb53e4f8dbf4567bd76dadfe3; no voters:
I20260810 00:50:52.146843 30030 leader_election.cc:288] T 00000000000000000000000000000000 P 5d52892bb53e4f8dbf4567bd76dadfe3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:50:52.147012 30044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d52892bb53e4f8dbf4567bd76dadfe3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:50:52.147826 30044 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d52892bb53e4f8dbf4567bd76dadfe3 [term 1 LEADER]: Becoming Leader. State: Replica: 5d52892bb53e4f8dbf4567bd76dadfe3, State: Running, Role: LEADER
I20260810 00:50:52.148381 30044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d52892bb53e4f8dbf4567bd76dadfe3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d52892bb53e4f8dbf4567bd76dadfe3" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 37049 } }
I20260810 00:50:52.149188 30030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d52892bb53e4f8dbf4567bd76dadfe3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:50:52.151300 30047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d52892bb53e4f8dbf4567bd76dadfe3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d52892bb53e4f8dbf4567bd76dadfe3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d52892bb53e4f8dbf4567bd76dadfe3" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 37049 } } }
I20260810 00:50:52.152413 30047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d52892bb53e4f8dbf4567bd76dadfe3 [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:52.151593 30049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d52892bb53e4f8dbf4567bd76dadfe3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d52892bb53e4f8dbf4567bd76dadfe3. Latest consensus state: current_term: 1 leader_uuid: "5d52892bb53e4f8dbf4567bd76dadfe3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d52892bb53e4f8dbf4567bd76dadfe3" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 37049 } } }
I20260810 00:50:52.152979 30049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d52892bb53e4f8dbf4567bd76dadfe3 [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:52.154350 30059 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:50:52.159567 30059 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:50:52.168416 30059 catalog_manager.cc:1347] Generated new cluster ID: 94303411e851402b8da1417f61641dbc
I20260810 00:50:52.168532 30059 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:50:52.195263 30059 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:50:52.195680 30059 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:50:52.213536 30059 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d52892bb53e4f8dbf4567bd76dadfe3: Generated new TSK 0
I20260810 00:50:52.214299 30059 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 00:50:52.228524 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 29908
W20260810 00:50:52.229666 30017 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44245
W20260810 00:50:52.230350 28696 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 29908: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 00:50:52.230485 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 29908
2026-08-10T00:50:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2214 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:50:52 dist-test-slave-1nl6 krb5kdc[30075](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:50:52 dist-test-slave-1nl6 krb5kdc[30075](info): set up 2 sockets
Aug 10 00:50:52 dist-test-slave-1nl6 krb5kdc[30075](info): commencing operation
krb5kdc: starting...
W20260810 00:50:53.550949 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.273s user 0.003s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:50:53 dist-test-slave-1nl6 krb5kdc[30075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:50:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:50:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.6.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
I20260810 00:50:53.849895 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:45981
--webserver_interface=127.28.6.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:39933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:45981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:50:54.196727 30100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:50:54.196986 30100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:50:54.197160 30100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:50:54.204305 30100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:50:54.204424 30100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:50:54.204528 30100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:50:54.204628 30100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:50:54.204705 30100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:50:54.214097 30100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:39933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:45981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:45981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:50:54.215415 30100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:50:54.216714 30100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:50:54.228569 30105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:54.228617 30106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:54.228524 30108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:50:54.234714 30100 server_base.cc:1034] running on GCE node
I20260810 00:50:54.235491 30100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:50:54.236800 30100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:50:54.238023 30100 hybrid_clock.cc:648] HybridClock initialized: now 1786323054237980 us; error 51 us; skew 500 ppm
I20260810 00:50:54.238354 30100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:50:54 dist-test-slave-1nl6 krb5kdc[30075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323054, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:50:54.241436 30100 init.cc:377] Logged in from keytab as kudu/127.28.6.62@KRBTEST.COM (short username kudu)
I20260810 00:50:54.243021 30100 webserver.cc:460] Webserver started at http://127.28.6.62:45341/ using document root <none> and password file <none>
I20260810 00:50:54.243680 30100 fs_manager.cc:359] Metadata directory not provided
I20260810 00:50:54.243830 30100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:50:54.244127 30100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:50:54.247000 30100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "4b313c72a74d4e19a6647986e4b0aa53"
format_stamp: "Formatted at 2026-08-10 00:50:54 on dist-test-slave-1nl6"
server_key: "b3750121d7adf2cd948a6240dd70d3c2"
server_key_iv: "d8635ecc6630dab1d0cf10ecd6af013e"
server_key_version: "encryptionkey@0"
I20260810 00:50:54.247757 30100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "4b313c72a74d4e19a6647986e4b0aa53"
format_stamp: "Formatted at 2026-08-10 00:50:54 on dist-test-slave-1nl6"
server_key: "b3750121d7adf2cd948a6240dd70d3c2"
server_key_iv: "d8635ecc6630dab1d0cf10ecd6af013e"
server_key_version: "encryptionkey@0"
I20260810 00:50:54.252555 30100 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 00:50:54.255765 30121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:54.257071 30100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 00:50:54.257277 30100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "4b313c72a74d4e19a6647986e4b0aa53"
format_stamp: "Formatted at 2026-08-10 00:50:54 on dist-test-slave-1nl6"
server_key: "b3750121d7adf2cd948a6240dd70d3c2"
server_key_iv: "d8635ecc6630dab1d0cf10ecd6af013e"
server_key_version: "encryptionkey@0"
I20260810 00:50:54.257474 30100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:50:54.281317 30100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:50:54.285154 30100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:50:54.285430 30100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:50:54.304008 30100 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:45981
I20260810 00:50:54.304237 30198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:45981 every 8 connection(s)
I20260810 00:50:54.305356 30100 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:50:54.308969 30199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:50:54.309449 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 30100
I20260810 00:50:54.309628 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:50:54.310041 28696 external_mini_cluster.cc:1428] Setting key 995f2b0bfd87d8e7bea0486af75af9e8
I20260810 00:50:54.322527 30199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b313c72a74d4e19a6647986e4b0aa53: Bootstrap starting.
Aug 10 00:50:54 dist-test-slave-1nl6 krb5kdc[30075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:50:54.326174 30199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b313c72a74d4e19a6647986e4b0aa53: Neither blocks nor log segments found. Creating new log.
I20260810 00:50:54.327286 30199 log.cc:824] T 00000000000000000000000000000000 P 4b313c72a74d4e19a6647986e4b0aa53: Log is configured to *not* fsync() on all Append() calls
I20260810 00:50:54.330039 30199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b313c72a74d4e19a6647986e4b0aa53: No bootstrap required, opened a new log
I20260810 00:50:54.333523 30202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:54.311760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44973 (local address 127.28.6.62:45981)
0810 00:50:54.312554 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:54.312565 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:54.312610 (+ 45us) server_negotiation.cc:407] Connection header received
0810 00:50:54.314717 (+ 2107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:54.314750 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:54.315178 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:54.315495 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:54.317284 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:54.318535 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:54.320441 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:54.321055 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:54.321175 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:54.326210 (+ 5035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:54.326219 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:54.326233 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:50:54.326266 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:54.329437 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:54.330169 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:54.330176 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:54.330182 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:54.330344 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:54.330768 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:54.330772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:54.330774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:54.331322 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:54.331674 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:54.332104 (+ 430us) server_negotiation.cc:299] Negotiation successful
0810 00:50:54.332542 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":398,"threads_started":1}
I20260810 00:50:54.337170 28696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 00:50:54.338143 30199 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b313c72a74d4e19a6647986e4b0aa53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b313c72a74d4e19a6647986e4b0aa53" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 45981 } }
I20260810 00:50:54.338480 30199 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b313c72a74d4e19a6647986e4b0aa53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:50:54.338586 30199 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b313c72a74d4e19a6647986e4b0aa53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b313c72a74d4e19a6647986e4b0aa53, State: Initialized, Role: FOLLOWER
I20260810 00:50:54.339206 30199 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b313c72a74d4e19a6647986e4b0aa53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b313c72a74d4e19a6647986e4b0aa53" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 45981 } }
I20260810 00:50:54.339491 30199 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b313c72a74d4e19a6647986e4b0aa53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:50:54.339641 30199 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b313c72a74d4e19a6647986e4b0aa53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:50:54.339851 30199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b313c72a74d4e19a6647986e4b0aa53 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:50:54.342577 30199 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b313c72a74d4e19a6647986e4b0aa53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b313c72a74d4e19a6647986e4b0aa53" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 45981 } }
I20260810 00:50:54.343150 30199 leader_election.cc:302] T 00000000000000000000000000000000 P 4b313c72a74d4e19a6647986e4b0aa53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b313c72a74d4e19a6647986e4b0aa53; no voters:
I20260810 00:50:54.343794 30199 leader_election.cc:288] T 00000000000000000000000000000000 P 4b313c72a74d4e19a6647986e4b0aa53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:50:54.343881 30213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b313c72a74d4e19a6647986e4b0aa53 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:50:54.346240 30199 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b313c72a74d4e19a6647986e4b0aa53 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:50:54.347188 30213 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b313c72a74d4e19a6647986e4b0aa53 [term 1 LEADER]: Becoming Leader. State: Replica: 4b313c72a74d4e19a6647986e4b0aa53, State: Running, Role: LEADER
I20260810 00:50:54.347905 30213 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b313c72a74d4e19a6647986e4b0aa53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b313c72a74d4e19a6647986e4b0aa53" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 45981 } }
I20260810 00:50:54.352135 30215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b313c72a74d4e19a6647986e4b0aa53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b313c72a74d4e19a6647986e4b0aa53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b313c72a74d4e19a6647986e4b0aa53" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 45981 } } }
I20260810 00:50:54.352859 30215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b313c72a74d4e19a6647986e4b0aa53 [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:54.352821 30216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b313c72a74d4e19a6647986e4b0aa53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b313c72a74d4e19a6647986e4b0aa53. Latest consensus state: current_term: 1 leader_uuid: "4b313c72a74d4e19a6647986e4b0aa53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b313c72a74d4e19a6647986e4b0aa53" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 45981 } } }
I20260810 00:50:54.353143 30216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b313c72a74d4e19a6647986e4b0aa53 [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:54.359540 30202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:54.341539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43683 (local address 127.28.6.62:45981)
0810 00:50:54.341727 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:54.341731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:54.342240 (+ 509us) server_negotiation.cc:407] Connection header received
0810 00:50:54.342412 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:54.342417 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:54.342486 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:54.342582 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:54.344294 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:54.345248 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:54.346662 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:54.347409 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:54.347566 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:54.350133 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:54.350142 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:54.350149 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:50:54.350206 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:54.354662 (+ 4456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:54.355792 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:54.355800 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:54.355806 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:54.355984 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:50:54.356547 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:50:54.356555 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:50:54.356561 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:50:54.357064 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:54.357342 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:54.358699 (+ 1357us) server_negotiation.cc:299] Negotiation successful
0810 00:50:54.358992 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":59}
I20260810 00:50:54.363936 30226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:50:54.368193 30226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:50:54.377099 30226 catalog_manager.cc:1347] Generated new cluster ID: c041b9e721424189842f92a6fbc16f6b
I20260810 00:50:54.377213 30226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:50:54.394443 30226 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:50:54.395805 30226 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:50:54.404270 30226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b313c72a74d4e19a6647986e4b0aa53: Generated new TSK 0
I20260810 00:50:54.405022 30226 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 00:50:54.416085 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 30100
I20260810 00:50:54.417313 30193 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:50:54.507169 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 30100
2026-08-10T00:50:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2271 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:50:54 dist-test-slave-1nl6 krb5kdc[30245](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:50:54 dist-test-slave-1nl6 krb5kdc[30245](info): set up 2 sockets
Aug 10 00:50:54 dist-test-slave-1nl6 krb5kdc[30245](info): commencing operation
krb5kdc: starting...
W20260810 00:50:55.820911 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.271s user 0.008s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:50:56 dist-test-slave-1nl6 krb5kdc[30245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:50:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:50:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.6.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
I20260810 00:50:56.125685 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:36003
--webserver_interface=127.28.6.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:36353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:36003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:50:56.468957 30265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:50:56.469316 30265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:50:56.469471 30265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:50:56.476603 30265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:50:56.476737 30265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:50:56.476833 30265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:50:56.476938 30265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:50:56.477017 30265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:50:56.486395 30265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:36353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:36003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:36003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:50:56.487649 30265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:50:56.488924 30265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:50:56.500808 30272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:56.500757 30274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:56.500855 30276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:50:56.501797 30265 server_base.cc:1034] running on GCE node
I20260810 00:50:56.502559 30265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:50:56.503988 30265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:50:56.505239 30265 hybrid_clock.cc:648] HybridClock initialized: now 1786323056505200 us; error 45 us; skew 500 ppm
I20260810 00:50:56.505589 30265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:50:56 dist-test-slave-1nl6 krb5kdc[30245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323056, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:50:56.512799 30265 init.cc:377] Logged in from keytab as kudu/127.28.6.62@KRBTEST.COM (short username kudu)
I20260810 00:50:56.514124 30265 webserver.cc:460] Webserver started at http://127.28.6.62:35817/ using document root <none> and password file <none>
I20260810 00:50:56.514748 30265 fs_manager.cc:359] Metadata directory not provided
I20260810 00:50:56.514849 30265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:50:56.515177 30265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:50:56.518065 30265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "a739a5875a4e4e73bed6613443d7d7c2"
format_stamp: "Formatted at 2026-08-10 00:50:56 on dist-test-slave-1nl6"
server_key: "484454d4eb154b45a97836c1a3bf1c3e"
server_key_iv: "f24877ec8d5d50bd97ce2c4a92fe0207"
server_key_version: "encryptionkey@0"
I20260810 00:50:56.518774 30265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "a739a5875a4e4e73bed6613443d7d7c2"
format_stamp: "Formatted at 2026-08-10 00:50:56 on dist-test-slave-1nl6"
server_key: "484454d4eb154b45a97836c1a3bf1c3e"
server_key_iv: "f24877ec8d5d50bd97ce2c4a92fe0207"
server_key_version: "encryptionkey@0"
I20260810 00:50:56.523933 30265 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260810 00:50:56.527256 30286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:56.528492 30265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 00:50:56.528673 30265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "a739a5875a4e4e73bed6613443d7d7c2"
format_stamp: "Formatted at 2026-08-10 00:50:56 on dist-test-slave-1nl6"
server_key: "484454d4eb154b45a97836c1a3bf1c3e"
server_key_iv: "f24877ec8d5d50bd97ce2c4a92fe0207"
server_key_version: "encryptionkey@0"
I20260810 00:50:56.528856 30265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:50:56.547348 30265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:50:56.551514 30265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:50:56.551801 30265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:50:56.568404 30265 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:36003
I20260810 00:50:56.568446 30359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:36003 every 8 connection(s)
I20260810 00:50:56.569648 30265 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:50:56.573220 30360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:50:56.578254 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 30265
I20260810 00:50:56.578426 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:50:56.578855 28696 external_mini_cluster.cc:1428] Setting key 626e7efec13f616f83521ceb89953614
I20260810 00:50:56.583777 30360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a739a5875a4e4e73bed6613443d7d7c2: Bootstrap starting.
I20260810 00:50:56.586740 30360 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a739a5875a4e4e73bed6613443d7d7c2: Neither blocks nor log segments found. Creating new log.
I20260810 00:50:56.587809 30360 log.cc:824] T 00000000000000000000000000000000 P a739a5875a4e4e73bed6613443d7d7c2: Log is configured to *not* fsync() on all Append() calls
I20260810 00:50:56.590447 30360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a739a5875a4e4e73bed6613443d7d7c2: No bootstrap required, opened a new log
I20260810 00:50:56.591507 30364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:56.580616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34275 (local address 127.28.6.62:36003)
0810 00:50:56.581507 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:56.581518 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:56.581562 (+ 44us) server_negotiation.cc:407] Connection header received
0810 00:50:56.583261 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:56.583284 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:56.583636 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:56.583998 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:56.584524 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:56.585799 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:56.587270 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:56.587919 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:56.588162 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:56.589305 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:56.589313 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:56.589326 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:56.589827 (+ 501us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:56.589900 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:56.589924 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:56.590054 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:56.590265 (+ 211us) server_negotiation.cc:299] Negotiation successful
0810 00:50:56.590525 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":401,"threads_started":1}
I20260810 00:50:56.595263 28696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 00:50:56.598855 30360 raft_consensus.cc:348] T 00000000000000000000000000000000 P a739a5875a4e4e73bed6613443d7d7c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a739a5875a4e4e73bed6613443d7d7c2" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36003 } }
I20260810 00:50:56.599337 30360 raft_consensus.cc:374] T 00000000000000000000000000000000 P a739a5875a4e4e73bed6613443d7d7c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:50:56.599469 30360 raft_consensus.cc:729] T 00000000000000000000000000000000 P a739a5875a4e4e73bed6613443d7d7c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a739a5875a4e4e73bed6613443d7d7c2, State: Initialized, Role: FOLLOWER
I20260810 00:50:56.600256 30360 consensus_queue.cc:260] T 00000000000000000000000000000000 P a739a5875a4e4e73bed6613443d7d7c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a739a5875a4e4e73bed6613443d7d7c2" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36003 } }
I20260810 00:50:56.600553 30360 raft_consensus.cc:388] T 00000000000000000000000000000000 P a739a5875a4e4e73bed6613443d7d7c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:50:56.600729 30360 raft_consensus.cc:482] T 00000000000000000000000000000000 P a739a5875a4e4e73bed6613443d7d7c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:50:56.600929 30360 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a739a5875a4e4e73bed6613443d7d7c2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:50:56.603974 30360 raft_consensus.cc:504] T 00000000000000000000000000000000 P a739a5875a4e4e73bed6613443d7d7c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a739a5875a4e4e73bed6613443d7d7c2" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36003 } }
I20260810 00:50:56.604545 30360 leader_election.cc:302] T 00000000000000000000000000000000 P a739a5875a4e4e73bed6613443d7d7c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a739a5875a4e4e73bed6613443d7d7c2; no voters:
I20260810 00:50:56.605266 30360 leader_election.cc:288] T 00000000000000000000000000000000 P a739a5875a4e4e73bed6613443d7d7c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:50:56.605368 30374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a739a5875a4e4e73bed6613443d7d7c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:50:56.606660 30364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:56.599437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54293 (local address 127.28.6.62:36003)
0810 00:50:56.599652 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:56.599661 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:56.600033 (+ 372us) server_negotiation.cc:407] Connection header received
0810 00:50:56.600242 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:56.600246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:56.600299 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:56.600375 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:56.601038 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:56.601774 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:56.603663 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:50:56.604177 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:50:56.604272 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:50:56.605537 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:56.605546 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:56.605552 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:56.605576 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:56.605626 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:56.605656 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:56.605778 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:56.606267 (+ 489us) server_negotiation.cc:299] Negotiation successful
0810 00:50:56.606351 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 00:50:56.607230 30374 raft_consensus.cc:686] T 00000000000000000000000000000000 P a739a5875a4e4e73bed6613443d7d7c2 [term 1 LEADER]: Becoming Leader. State: Replica: a739a5875a4e4e73bed6613443d7d7c2, State: Running, Role: LEADER
I20260810 00:50:56.607795 30360 sys_catalog.cc:564] T 00000000000000000000000000000000 P a739a5875a4e4e73bed6613443d7d7c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:50:56.607853 30374 consensus_queue.cc:237] T 00000000000000000000000000000000 P a739a5875a4e4e73bed6613443d7d7c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a739a5875a4e4e73bed6613443d7d7c2" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36003 } }
I20260810 00:50:56.610602 30380 sys_catalog.cc:455] T 00000000000000000000000000000000 P a739a5875a4e4e73bed6613443d7d7c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a739a5875a4e4e73bed6613443d7d7c2. Latest consensus state: current_term: 1 leader_uuid: "a739a5875a4e4e73bed6613443d7d7c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a739a5875a4e4e73bed6613443d7d7c2" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36003 } } }
I20260810 00:50:56.611002 30380 sys_catalog.cc:458] T 00000000000000000000000000000000 P a739a5875a4e4e73bed6613443d7d7c2 [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:56.611655 30377 sys_catalog.cc:455] T 00000000000000000000000000000000 P a739a5875a4e4e73bed6613443d7d7c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a739a5875a4e4e73bed6613443d7d7c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a739a5875a4e4e73bed6613443d7d7c2" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 36003 } } }
I20260810 00:50:56.611939 30377 sys_catalog.cc:458] T 00000000000000000000000000000000 P a739a5875a4e4e73bed6613443d7d7c2 [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:56.614373 30390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:50:56.618522 30390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:50:56.627252 30390 catalog_manager.cc:1347] Generated new cluster ID: d4480e71e6eb4459b1123d35cb53aac6
I20260810 00:50:56.627393 30390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:50:56.651221 30390 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:50:56.652456 30390 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:50:56.658468 30390 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a739a5875a4e4e73bed6613443d7d7c2: Generated new TSK 0
I20260810 00:50:56.659499 30390 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 00:50:56.676668 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 30265
W20260810 00:50:56.678243 30352 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34275
W20260810 00:50:56.678936 28696 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 30265: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 00:50:56.679095 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 30265
2026-08-10T00:50:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2176 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:50:56 dist-test-slave-1nl6 krb5kdc[30407](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:50:56 dist-test-slave-1nl6 krb5kdc[30407](info): set up 2 sockets
Aug 10 00:50:56 dist-test-slave-1nl6 krb5kdc[30407](info): commencing operation
krb5kdc: starting...
W20260810 00:50:58.007593 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.282s user 0.007s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:50:58 dist-test-slave-1nl6 krb5kdc[30407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:50:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:50:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.6.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
I20260810 00:50:58.301687 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:37719
--webserver_interface=127.28.6.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:37529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:37719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:50:58.637786 30426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:50:58.638074 30426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:50:58.638271 30426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:50:58.645507 30426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:50:58.645617 30426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:50:58.645744 30426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:50:58.645860 30426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:50:58.645923 30426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:50:58.655797 30426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:37529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:37719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:37719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:50:58.657114 30426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:50:58.658428 30426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:50:58.670377 30432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:58.670614 30434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:50:58.670866 30437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:50:58.671674 30426 server_base.cc:1034] running on GCE node
I20260810 00:50:58.672626 30426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:50:58.674295 30426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:50:58.675534 30426 hybrid_clock.cc:648] HybridClock initialized: now 1786323058675486 us; error 43 us; skew 500 ppm
I20260810 00:50:58.675902 30426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:50:58 dist-test-slave-1nl6 krb5kdc[30407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323058, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:50:58.680187 30426 init.cc:377] Logged in from keytab as kudu/127.28.6.62@KRBTEST.COM (short username kudu)
I20260810 00:50:58.681723 30426 webserver.cc:460] Webserver started at http://127.28.6.62:36863/ using document root <none> and password file <none>
I20260810 00:50:58.682410 30426 fs_manager.cc:359] Metadata directory not provided
I20260810 00:50:58.682560 30426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:50:58.682875 30426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:50:58.685885 30426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "80b5bd6fea9848e6999ce7a4dfee2e4a"
format_stamp: "Formatted at 2026-08-10 00:50:58 on dist-test-slave-1nl6"
server_key: "73d69ec8fb4829453704cbb5d54d0847"
server_key_iv: "3e1327cf525e1a782e600fcbb5a63880"
server_key_version: "encryptionkey@0"
I20260810 00:50:58.686666 30426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "80b5bd6fea9848e6999ce7a4dfee2e4a"
format_stamp: "Formatted at 2026-08-10 00:50:58 on dist-test-slave-1nl6"
server_key: "73d69ec8fb4829453704cbb5d54d0847"
server_key_iv: "3e1327cf525e1a782e600fcbb5a63880"
server_key_version: "encryptionkey@0"
I20260810 00:50:58.691865 30426 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260810 00:50:58.695221 30445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:50:58.696527 30426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 00:50:58.696815 30426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "80b5bd6fea9848e6999ce7a4dfee2e4a"
format_stamp: "Formatted at 2026-08-10 00:50:58 on dist-test-slave-1nl6"
server_key: "73d69ec8fb4829453704cbb5d54d0847"
server_key_iv: "3e1327cf525e1a782e600fcbb5a63880"
server_key_version: "encryptionkey@0"
I20260810 00:50:58.697014 30426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:50:58.702841 30426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:50:58.706894 30426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:50:58.707254 30426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:50:58.726358 30426 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:37719
I20260810 00:50:58.726447 30523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:37719 every 8 connection(s)
I20260810 00:50:58.727851 30426 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:50:58.731643 30525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:50:58.737840 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 30426
I20260810 00:50:58.738040 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:50:58.738606 28696 external_mini_cluster.cc:1428] Setting key 59fcb4e2d162036f1d2ee19fff67226d
I20260810 00:50:58.742938 30525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80b5bd6fea9848e6999ce7a4dfee2e4a: Bootstrap starting.
I20260810 00:50:58.746467 30528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:58.740498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50507 (local address 127.28.6.62:37719)
0810 00:50:58.741374 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:58.741386 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:58.741428 (+ 42us) server_negotiation.cc:407] Connection header received
0810 00:50:58.743185 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:58.743208 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:58.743533 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:58.743827 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:58.744973 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:58.744979 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:58.744992 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:58.745006 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:58.745070 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:58.745089 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:58.745264 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:58.745415 (+ 151us) server_negotiation.cc:299] Negotiation successful
0810 00:50:58.745618 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":414,"threads_started":1}
I20260810 00:50:58.746560 30525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80b5bd6fea9848e6999ce7a4dfee2e4a: Neither blocks nor log segments found. Creating new log.
I20260810 00:50:58.747932 30525 log.cc:824] T 00000000000000000000000000000000 P 80b5bd6fea9848e6999ce7a4dfee2e4a: Log is configured to *not* fsync() on all Append() calls
I20260810 00:50:58.750352 28696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 00:50:58.751164 30525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80b5bd6fea9848e6999ce7a4dfee2e4a: No bootstrap required, opened a new log
I20260810 00:50:58.757588 30528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:50:58.754921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55005 (local address 127.28.6.62:37719)
0810 00:50:58.755122 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 00:50:58.755129 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:50:58.755678 (+ 549us) server_negotiation.cc:407] Connection header received
0810 00:50:58.755835 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:50:58.755839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:50:58.755887 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:50:58.755966 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:50:58.756926 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:50:58.756930 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:50:58.756933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 00:50:58.756947 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:50:58.756988 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 00:50:58.757006 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:50:58.757106 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 00:50:58.757236 (+ 130us) server_negotiation.cc:299] Negotiation successful
0810 00:50:58.757283 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 00:50:58.760170 30525 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80b5bd6fea9848e6999ce7a4dfee2e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80b5bd6fea9848e6999ce7a4dfee2e4a" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 37719 } }
I20260810 00:50:58.760517 30525 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80b5bd6fea9848e6999ce7a4dfee2e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 00:50:58.760636 30525 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80b5bd6fea9848e6999ce7a4dfee2e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80b5bd6fea9848e6999ce7a4dfee2e4a, State: Initialized, Role: FOLLOWER
I20260810 00:50:58.761186 30525 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80b5bd6fea9848e6999ce7a4dfee2e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80b5bd6fea9848e6999ce7a4dfee2e4a" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 37719 } }
I20260810 00:50:58.761427 30525 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80b5bd6fea9848e6999ce7a4dfee2e4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 00:50:58.761579 30525 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80b5bd6fea9848e6999ce7a4dfee2e4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 00:50:58.761767 30525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80b5bd6fea9848e6999ce7a4dfee2e4a [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:50:58.764135 30525 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80b5bd6fea9848e6999ce7a4dfee2e4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80b5bd6fea9848e6999ce7a4dfee2e4a" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 37719 } }
I20260810 00:50:58.764705 30525 leader_election.cc:302] T 00000000000000000000000000000000 P 80b5bd6fea9848e6999ce7a4dfee2e4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80b5bd6fea9848e6999ce7a4dfee2e4a; no voters:
I20260810 00:50:58.765476 30525 leader_election.cc:288] T 00000000000000000000000000000000 P 80b5bd6fea9848e6999ce7a4dfee2e4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 00:50:58.765651 30536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80b5bd6fea9848e6999ce7a4dfee2e4a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:50:58.766425 30536 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80b5bd6fea9848e6999ce7a4dfee2e4a [term 1 LEADER]: Becoming Leader. State: Replica: 80b5bd6fea9848e6999ce7a4dfee2e4a, State: Running, Role: LEADER
I20260810 00:50:58.766992 30536 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80b5bd6fea9848e6999ce7a4dfee2e4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80b5bd6fea9848e6999ce7a4dfee2e4a" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 37719 } }
I20260810 00:50:58.767760 30525 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80b5bd6fea9848e6999ce7a4dfee2e4a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:50:58.769629 30538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80b5bd6fea9848e6999ce7a4dfee2e4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80b5bd6fea9848e6999ce7a4dfee2e4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80b5bd6fea9848e6999ce7a4dfee2e4a" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 37719 } } }
I20260810 00:50:58.770062 30538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80b5bd6fea9848e6999ce7a4dfee2e4a [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:58.776134 30550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:50:58.775929 30539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80b5bd6fea9848e6999ce7a4dfee2e4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80b5bd6fea9848e6999ce7a4dfee2e4a. Latest consensus state: current_term: 1 leader_uuid: "80b5bd6fea9848e6999ce7a4dfee2e4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80b5bd6fea9848e6999ce7a4dfee2e4a" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 37719 } } }
I20260810 00:50:58.776404 30539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80b5bd6fea9848e6999ce7a4dfee2e4a [sys.catalog]: This master's current role is: LEADER
I20260810 00:50:58.780686 30550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:50:58.789335 30550 catalog_manager.cc:1347] Generated new cluster ID: 1b5b5639b72941aea65385ed11ade04a
I20260810 00:50:58.789438 30550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:50:58.803015 30550 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:50:58.803530 30550 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:50:58.814083 30550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80b5bd6fea9848e6999ce7a4dfee2e4a: Generated new TSK 0
I20260810 00:50:58.815040 30550 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 00:50:58.824951 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 30426
W20260810 00:50:58.826040 30518 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50507
W20260810 00:50:58.826757 28696 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 30426: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 00:50:58.826931 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 30426
2026-08-10T00:50:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2145 ms)
[----------] 18 tests from AuthTokenIssuingTest (39629 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:50:58 dist-test-slave-1nl6 krb5kdc[30570](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:50:58 dist-test-slave-1nl6 krb5kdc[30570](info): set up 2 sockets
Aug 10 00:50:58 dist-test-slave-1nl6 krb5kdc[30570](info): commencing operation
krb5kdc: starting...
W20260810 00:51:00.162207 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.289s user 0.000s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:51:00 dist-test-slave-1nl6 krb5kdc[30570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:51:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:51:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.6.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
I20260810 00:51:00.485922 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:42159
--webserver_interface=127.28.6.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:45377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:42159,127.28.6.61:35219,127.28.6.60:36819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:51:00.825093 30588 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 00:51:00.825475 30588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:51:00.825606 30588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:51:00.825747 30588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:51:00.832789 30588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:51:00.832914 30588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:51:00.833025 30588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:51:00.833113 30588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:51:00.833205 30588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:51:00.842480 30588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:45377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:42159,127.28.6.61:35219,127.28.6.60:36819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:42159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:51:00.843794 30588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:51:00.845131 30588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:51:00.857101 30596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:51:00.857141 30594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:51:00.857103 30600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:51:00.858388 30588 server_base.cc:1034] running on GCE node
I20260810 00:51:00.859236 30588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:51:00.860507 30588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:51:00.861764 30588 hybrid_clock.cc:648] HybridClock initialized: now 1786323060861708 us; error 41 us; skew 500 ppm
I20260810 00:51:00.862087 30588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:51:00 dist-test-slave-1nl6 krb5kdc[30570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323060, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:51:00.865171 30588 init.cc:377] Logged in from keytab as kudu/127.28.6.62@KRBTEST.COM (short username kudu)
I20260810 00:51:00.866459 30588 webserver.cc:460] Webserver started at http://127.28.6.62:34589/ using document root <none> and password file <none>
I20260810 00:51:00.867154 30588 fs_manager.cc:359] Metadata directory not provided
I20260810 00:51:00.867244 30588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:51:00.867507 30588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:51:00.870287 30588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "4e478568994b463094671ad36417c3d3"
format_stamp: "Formatted at 2026-08-10 00:51:00 on dist-test-slave-1nl6"
server_key: "c4cbc418438828dfbca326ba0cd3aca6"
server_key_iv: "d68cbb4b1cdff78f6957cdba2f079612"
server_key_version: "encryptionkey@0"
I20260810 00:51:00.871116 30588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "4e478568994b463094671ad36417c3d3"
format_stamp: "Formatted at 2026-08-10 00:51:00 on dist-test-slave-1nl6"
server_key: "c4cbc418438828dfbca326ba0cd3aca6"
server_key_iv: "d68cbb4b1cdff78f6957cdba2f079612"
server_key_version: "encryptionkey@0"
I20260810 00:51:00.875984 30588 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 00:51:00.879216 30608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:51:00.880446 30588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 00:51:00.880584 30588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "4e478568994b463094671ad36417c3d3"
format_stamp: "Formatted at 2026-08-10 00:51:00 on dist-test-slave-1nl6"
server_key: "c4cbc418438828dfbca326ba0cd3aca6"
server_key_iv: "d68cbb4b1cdff78f6957cdba2f079612"
server_key_version: "encryptionkey@0"
I20260810 00:51:00.880764 30588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:51:00.899585 30588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:51:00.903401 30588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:51:00.903620 30588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:51:00.922264 30588 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:42159
I20260810 00:51:00.922276 30679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:42159 every 8 connection(s)
I20260810 00:51:00.923727 30588 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:51:00.927445 30683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:51:00.934000 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 30588
I20260810 00:51:00.934231 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:51:00.933854 30683 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.6.62" port: 42159 } has no permanent_uuid. Determining permanent_uuid...
I20260810 00:51:00.934947 28696 external_mini_cluster.cc:1428] Setting key eee1ee3269a202f596890c9026f9868c
Aug 10 00:51:00 dist-test-slave-1nl6 krb5kdc[30570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323060, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:51:00.955756 30688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:00.938423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45524 (local address 127.28.6.62:42159)
0810 00:51:00.939160 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:00.939173 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:00.939223 (+ 50us) server_negotiation.cc:407] Connection header received
0810 00:51:00.940344 (+ 1121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:00.940349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:00.940568 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:00.940778 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:00.941989 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:00.943245 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:00.944573 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:00.945100 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:00.945302 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:00.948635 (+ 3333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:51:00.948643 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:51:00.948653 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:51:00.948679 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:51:00.951541 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:00.952441 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:00.952447 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:00.952450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:00.952593 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:00.953059 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:00.953063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:00.953065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:00.953672 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:51:00.953834 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:00.954474 (+ 640us) server_negotiation.cc:299] Negotiation successful
0810 00:51:00.954730 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":521,"thread_start_us":332,"threads_started":1}
I20260810 00:51:00.955796 30685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:00.937149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:42159 (local address 127.0.0.1:45524)
0810 00:51:00.937922 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:00.937981 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 00:51:00.939997 (+ 2016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:51:00.940783 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:51:00.940789 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:51:00.941459 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:51:00.941778 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:51:00.941794 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:51:00.943392 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:51:00.943398 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:51:00.944432 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:51:00.944445 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:51:00.945331 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:51:00.945338 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:51:00.945601 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:00.946233 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:51:00.946261 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:51:00.948420 (+ 2159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:51:00.951674 (+ 3254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:51:00.951690 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:51:00.951703 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:51:00.952311 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:51:00.952717 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:51:00.952725 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:51:00.952731 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:51:00.952951 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:51:00.953835 (+ 884us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:51:00.953845 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:51:00.954094 (+ 249us) client_negotiation.cc:769] Sending connection context
0810 00:51:00.954374 (+ 280us) client_negotiation.cc:240] Negotiation successful
0810 00:51:00.954948 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"mutex_wait_us":74,"thread_start_us":347,"threads_started":1}
I20260810 00:51:00.958781 30683 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.6.61" port: 35219 } has no permanent_uuid. Determining permanent_uuid...
I20260810 00:51:00.959919 30685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:00.959433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:35219 (local address 127.0.0.1:52496)
0810 00:51:00.959628 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:00.959773 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.61:35219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260810 00:51:00.960163 30610 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.6.61:35219: connect: Connection refused (error 111)
I20260810 00:51:00.961802 30685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:00.961348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:35219 (local address 127.0.0.1:52498)
0810 00:51:00.961521 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:00.961620 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.61:35219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260810 00:51:00.962001 30683 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.61:35219: Network error: Client connection negotiation failed: client connection to 127.28.6.61:35219: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.28.6.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.61.keytab.
Entry for principal kudu/127.28.6.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.61.keytab.
I20260810 00:51:01.004127 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.61:35219
--webserver_interface=127.28.6.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.61.keytab
--principal=kudu/127.28.6.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:45377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:42159,127.28.6.61:35219,127.28.6.60:36819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 00:51:01.011363 30683 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.61" port: 35219 } attempt: 1
I20260810 00:51:01.012554 30685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:01.012095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:35219 (local address 127.0.0.1:52510)
0810 00:51:01.012311 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:01.012425 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.61:35219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260810 00:51:01.013537 30685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:01.013183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:35219 (local address 127.0.0.1:52512)
0810 00:51:01.013330 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:01.013404 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.61:35219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 00:51:01.013734 30683 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.61:35219: Network error: Client connection negotiation failed: client connection to 127.28.6.61:35219: connect: Connection refused (error 111)
I20260810 00:51:01.082003 30683 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.61" port: 35219 } attempt: 2
I20260810 00:51:01.083317 30685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:01.082707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:35219 (local address 127.0.0.1:52520)
0810 00:51:01.083020 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:01.083139 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.61:35219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260810 00:51:01.084478 30685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:01.084072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:35219 (local address 127.0.0.1:52524)
0810 00:51:01.084241 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:01.084325 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.61:35219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 00:51:01.084767 30683 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.61:35219: Network error: Client connection negotiation failed: client connection to 127.28.6.61:35219: connect: Connection refused (error 111)
I20260810 00:51:01.176060 30683 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.61" port: 35219 } attempt: 3
I20260810 00:51:01.177294 30685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:01.176767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:35219 (local address 127.0.0.1:52532)
0810 00:51:01.177006 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:01.177121 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.61:35219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260810 00:51:01.178419 30685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:01.178016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:35219 (local address 127.0.0.1:52540)
0810 00:51:01.178167 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:01.178253 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.61:35219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 00:51:01.178683 30683 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.61:35219: Network error: Client connection negotiation failed: client connection to 127.28.6.61:35219: connect: Connection refused (error 111)
I20260810 00:51:01.321974 30683 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.61" port: 35219 } attempt: 4
I20260810 00:51:01.323263 30685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:01.322626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:35219 (local address 127.0.0.1:52556)
0810 00:51:01.322984 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:01.323097 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.61:35219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260810 00:51:01.324425 30685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:01.324016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:35219 (local address 127.0.0.1:52560)
0810 00:51:01.324175 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:01.324278 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.61:35219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 00:51:01.324630 30683 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.61:35219: Network error: Client connection negotiation failed: client connection to 127.28.6.61:35219: connect: Connection refused (error 111)
W20260810 00:51:01.338006 30693 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 00:51:01.338342 30693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:51:01.338445 30693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:51:01.338577 30693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:51:01.345546 30693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:51:01.345619 30693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:51:01.345731 30693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:51:01.345810 30693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:51:01.345911 30693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:51:01.355280 30693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:45377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:42159,127.28.6.61:35219,127.28.6.60:36819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.61.keytab
--principal=kudu/127.28.6.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.61:35219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.6.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:51:01.356441 30693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:51:01.357740 30693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:51:01.368863 30700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:51:01.368963 30701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:51:01.368913 30706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:51:01.369882 30693 server_base.cc:1034] running on GCE node
I20260810 00:51:01.370611 30693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:51:01.371892 30693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:51:01.373126 30693 hybrid_clock.cc:648] HybridClock initialized: now 1786323061373030 us; error 67 us; skew 500 ppm
I20260810 00:51:01.373452 30693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:51:01 dist-test-slave-1nl6 krb5kdc[30570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323061, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:51:01.376602 30693 init.cc:377] Logged in from keytab as kudu/127.28.6.61@KRBTEST.COM (short username kudu)
I20260810 00:51:01.377930 30693 webserver.cc:460] Webserver started at http://127.28.6.61:40275/ using document root <none> and password file <none>
I20260810 00:51:01.378539 30693 fs_manager.cc:359] Metadata directory not provided
I20260810 00:51:01.378624 30693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:51:01.379041 30693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:51:01.381840 30693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-1/data/instance:
uuid: "0839701233a94e9092be66da3197ddc4"
format_stamp: "Formatted at 2026-08-10 00:51:01 on dist-test-slave-1nl6"
server_key: "ea0b7dbe16663f050fca65753a237361"
server_key_iv: "2f96808bbd2d7675148b855801bfaf89"
server_key_version: "encryptionkey@0"
I20260810 00:51:01.382642 30693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-1/wal/instance:
uuid: "0839701233a94e9092be66da3197ddc4"
format_stamp: "Formatted at 2026-08-10 00:51:01 on dist-test-slave-1nl6"
server_key: "ea0b7dbe16663f050fca65753a237361"
server_key_iv: "2f96808bbd2d7675148b855801bfaf89"
server_key_version: "encryptionkey@0"
I20260810 00:51:01.387524 30693 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 00:51:01.390744 30714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:51:01.392055 30693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260810 00:51:01.392257 30693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-1/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-1/wal
uuid: "0839701233a94e9092be66da3197ddc4"
format_stamp: "Formatted at 2026-08-10 00:51:01 on dist-test-slave-1nl6"
server_key: "ea0b7dbe16663f050fca65753a237361"
server_key_iv: "2f96808bbd2d7675148b855801bfaf89"
server_key_version: "encryptionkey@0"
I20260810 00:51:01.392459 30693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:51:01.421047 30693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:51:01.425320 30693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:51:01.425607 30693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:51:01.443992 30693 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.61:35219
I20260810 00:51:01.444012 30793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.61:35219 every 8 connection(s)
I20260810 00:51:01.445287 30693 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-1/data/info.pb
I20260810 00:51:01.449048 30795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:51:01.453434 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 30693
I20260810 00:51:01.453622 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-1/wal/instance
I20260810 00:51:01.454171 28696 external_mini_cluster.cc:1428] Setting key c02157943c4c152f25e04f5f1009594b
I20260810 00:51:01.456465 30795 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.6.62" port: 42159 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 00:51:01 dist-test-slave-1nl6 krb5kdc[30570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323061, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.61@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:51:01.479270 30798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:01.459650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45528 (local address 127.28.6.62:42159)
0810 00:51:01.460154 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:01.460163 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:01.461105 (+ 942us) server_negotiation.cc:407] Connection header received
0810 00:51:01.462958 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:01.462966 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:01.463058 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:01.463186 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:01.464617 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:01.465736 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:01.467864 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:01.468620 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:01.468770 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:01.473133 (+ 4363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:51:01.473138 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:51:01.473142 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:51:01.473185 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:51:01.475957 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:01.476920 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:01.476924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:01.476926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:01.477049 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:01.477514 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:01.477517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:01.477519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:01.477760 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:51:01.477880 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:01.478779 (+ 899us) server_negotiation.cc:299] Negotiation successful
0810 00:51:01.478995 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":246,"threads_started":1}
I20260810 00:51:01.480736 30800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:01.460175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:42159 (local address 127.0.0.1:45528)
0810 00:51:01.460932 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:01.460983 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 00:51:01.462662 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:51:01.463374 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:51:01.463381 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:51:01.463857 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:51:01.464399 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:51:01.464416 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:51:01.465904 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:51:01.465911 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:51:01.467711 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:51:01.467727 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:51:01.468870 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:51:01.468877 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:51:01.469191 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:01.469917 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:51:01.469956 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:51:01.472915 (+ 2959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:51:01.476101 (+ 3186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:51:01.476112 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:51:01.476128 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:51:01.476799 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:51:01.477158 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:51:01.477166 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:51:01.477172 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:51:01.477406 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:51:01.477898 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:51:01.477908 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:51:01.478372 (+ 464us) client_negotiation.cc:769] Sending connection context
0810 00:51:01.478716 (+ 344us) client_negotiation.cc:240] Negotiation successful
0810 00:51:01.479259 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":331,"threads_started":1}
I20260810 00:51:01.482477 30795 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.6.61" port: 35219 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.6.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.60@KRBTEST.COM" created.
Aug 10 00:51:01 dist-test-slave-1nl6 krb5kdc[30570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323061, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.61@KRBTEST.COM for kudu/127.28.6.61@KRBTEST.COM
I20260810 00:51:01.500111 30800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:01.483190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:35219 (local address 127.0.0.1:52564)
0810 00:51:01.483467 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:01.483488 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 00:51:01.483588 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:51:01.484593 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:51:01.484601 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:51:01.485120 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:51:01.485354 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:51:01.485370 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:51:01.486864 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:51:01.486872 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:51:01.488420 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:51:01.488441 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:51:01.489263 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:51:01.489275 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:51:01.489503 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:01.490105 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:51:01.490129 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:51:01.493865 (+ 3736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:51:01.497637 (+ 3772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:51:01.497642 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:51:01.497646 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:51:01.497975 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:51:01.498364 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:51:01.498370 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:51:01.498373 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:51:01.498521 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:51:01.499167 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:51:01.499171 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:51:01.499324 (+ 153us) client_negotiation.cc:769] Sending connection context
0810 00:51:01.499463 (+ 139us) client_negotiation.cc:240] Negotiation successful
0810 00:51:01.499725 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260810 00:51:01.500214 30802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:01.483415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52564 (local address 127.28.6.61:35219)
0810 00:51:01.484101 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:01.484107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:01.484172 (+ 65us) server_negotiation.cc:407] Connection header received
0810 00:51:01.484227 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:01.484232 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:01.484416 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:01.484543 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:01.485583 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:01.486686 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:01.488569 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:01.489091 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:01.489188 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:01.494094 (+ 4906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:51:01.494110 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:51:01.494130 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:51:01.494174 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:51:01.497503 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:01.498108 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:01.498116 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:01.498120 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:01.498255 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:01.498635 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:01.498639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:01.498643 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:01.499047 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:51:01.499254 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:01.499614 (+ 360us) server_negotiation.cc:299] Negotiation successful
0810 00:51:01.499829 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":389,"threads_started":1}
I20260810 00:51:01.502951 30795 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.6.60" port: 36819 } has no permanent_uuid. Determining permanent_uuid...
I20260810 00:51:01.504192 30800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:01.503593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:36819 (local address 127.0.0.1:43096)
0810 00:51:01.503802 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:01.503976 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:36819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260810 00:51:01.504446 30716 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.6.60:36819: connect: Connection refused (error 111)
I20260810 00:51:01.506707 30800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:01.506376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:36819 (local address 127.0.0.1:43112)
0810 00:51:01.506527 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:01.506611 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:36819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260810 00:51:01.506944 30795 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.60:36819: Network error: Client connection negotiation failed: client connection to 127.28.6.60:36819: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.60.keytab.
Entry for principal kudu/127.28.6.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.60.keytab.
I20260810 00:51:01.529752 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.60:36819
--webserver_interface=127.28.6.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.60.keytab
--principal=kudu/127.28.6.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:45377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:42159,127.28.6.61:35219,127.28.6.60:36819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 00:51:01.556257 30795 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.60" port: 36819 } attempt: 1
I20260810 00:51:01.557413 30800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:01.556959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:36819 (local address 127.0.0.1:43118)
0810 00:51:01.557162 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:01.557260 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:36819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260810 00:51:01.558637 30800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:01.558184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:36819 (local address 127.0.0.1:43124)
0810 00:51:01.558466 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:01.558537 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:36819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
W20260810 00:51:01.558858 30795 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.60:36819: Network error: Client connection negotiation failed: client connection to 127.28.6.60:36819: connect: Connection refused (error 111)
I20260810 00:51:01.623886 30683 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.61" port: 35219 } attempt: 5
I20260810 00:51:01.627123 30795 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.60" port: 36819 } attempt: 2
I20260810 00:51:01.628211 30800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:01.627710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:36819 (local address 127.0.0.1:43132)
0810 00:51:01.627912 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:01.628012 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:36819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260810 00:51:01.629384 30800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:01.628926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:36819 (local address 127.0.0.1:43140)
0810 00:51:01.629109 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:01.629201 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:36819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260810 00:51:01.629714 30795 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.60:36819: Network error: Client connection negotiation failed: client connection to 127.28.6.60:36819: connect: Connection refused (error 111)
Aug 10 00:51:01 dist-test-slave-1nl6 krb5kdc[30570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323060, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for kudu/127.28.6.61@KRBTEST.COM
I20260810 00:51:01.641268 30802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:01.624809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52580 (local address 127.28.6.61:35219)
0810 00:51:01.624979 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:01.624987 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:01.625014 (+ 27us) server_negotiation.cc:407] Connection header received
0810 00:51:01.625258 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:01.625263 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:01.625346 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:01.625457 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:01.626298 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:01.627152 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:01.628848 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:01.629513 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:01.629633 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:01.636075 (+ 6442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:51:01.636080 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:51:01.636083 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:51:01.636116 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:51:01.638663 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:01.639422 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:01.639429 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:01.639432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:01.639553 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:01.639941 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:01.639947 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:01.639950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:01.640244 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:51:01.640396 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:01.640782 (+ 386us) server_negotiation.cc:299] Negotiation successful
0810 00:51:01.640964 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 00:51:01.641307 30685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:01.624698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:35219 (local address 127.0.0.1:52580)
0810 00:51:01.624938 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:01.624959 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 00:51:01.625100 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:51:01.625496 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:51:01.625500 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:51:01.625928 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:51:01.626104 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:51:01.626114 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:51:01.627309 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:51:01.627316 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:51:01.628634 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:51:01.628661 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:51:01.629648 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:51:01.629654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:51:01.629861 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:01.633766 (+ 3905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:51:01.633784 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:51:01.635866 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:51:01.638800 (+ 2934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:51:01.638805 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:51:01.638810 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:51:01.639302 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:51:01.639672 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:51:01.639677 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:51:01.639680 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:51:01.639829 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:51:01.640367 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:51:01.640372 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:51:01.640512 (+ 140us) client_negotiation.cc:769] Sending connection context
0810 00:51:01.640690 (+ 178us) client_negotiation.cc:240] Negotiation successful
0810 00:51:01.640955 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260810 00:51:01.642668 30683 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.6.60" port: 36819 } has no permanent_uuid. Determining permanent_uuid...
I20260810 00:51:01.643688 30685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:01.643326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:36819 (local address 127.0.0.1:43144)
0810 00:51:01.643492 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:01.643561 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:36819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260810 00:51:01.644639 30685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:01.644328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:36819 (local address 127.0.0.1:43156)
0810 00:51:01.644476 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:01.644549 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:36819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 00:51:01.644868 30683 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.60:36819: Network error: Client connection negotiation failed: client connection to 127.28.6.60:36819: connect: Connection refused (error 111)
I20260810 00:51:01.696122 30683 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.60" port: 36819 } attempt: 1
I20260810 00:51:01.697517 30685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:01.697023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:36819 (local address 127.0.0.1:43166)
0810 00:51:01.697256 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:01.697347 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:36819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260810 00:51:01.698553 30685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:01.698237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:36819 (local address 127.0.0.1:43178)
0810 00:51:01.698390 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:01.698455 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:36819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 00:51:01.698758 30683 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.60:36819: Network error: Client connection negotiation failed: client connection to 127.28.6.60:36819: connect: Connection refused (error 111)
I20260810 00:51:01.720968 30795 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.60" port: 36819 } attempt: 3
I20260810 00:51:01.722312 30800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:01.721722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:36819 (local address 127.0.0.1:43192)
0810 00:51:01.721988 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:01.722107 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:36819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260810 00:51:01.723492 30800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:01.723030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:36819 (local address 127.0.0.1:43208)
0810 00:51:01.723259 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:01.723353 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:36819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260810 00:51:01.723733 30795 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.60:36819: Network error: Client connection negotiation failed: client connection to 127.28.6.60:36819: connect: Connection refused (error 111)
I20260810 00:51:01.766991 30683 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.60" port: 36819 } attempt: 2
I20260810 00:51:01.768548 30685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:01.767796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:36819 (local address 127.0.0.1:43224)
0810 00:51:01.768203 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:01.768322 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:36819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203}
I20260810 00:51:01.769913 30685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:01.769492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:36819 (local address 127.0.0.1:43236)
0810 00:51:01.769681 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:01.769776 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:36819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260810 00:51:01.770103 30683 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.60:36819: Network error: Client connection negotiation failed: client connection to 127.28.6.60:36819: connect: Connection refused (error 111)
I20260810 00:51:01.866994 30795 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.60" port: 36819 } attempt: 4
I20260810 00:51:01.868274 30800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:01.867695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:36819 (local address 127.0.0.1:43252)
0810 00:51:01.867947 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:01.868055 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:36819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260810 00:51:01.869431 30800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:01.868931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:36819 (local address 127.0.0.1:43258)
0810 00:51:01.869085 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:01.869169 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:36819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 00:51:01.869661 30795 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.60:36819: Network error: Client connection negotiation failed: client connection to 127.28.6.60:36819: connect: Connection refused (error 111)
I20260810 00:51:01.876314 30683 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.60" port: 36819 } attempt: 3
I20260810 00:51:01.877264 30685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:01.876877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:36819 (local address 127.0.0.1:43260)
0810 00:51:01.877051 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:01.877126 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:36819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260810 00:51:01.878197 30685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:01.877865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:36819 (local address 127.0.0.1:43272)
0810 00:51:01.878044 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:01.878108 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:36819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260810 00:51:01.878434 30683 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.60:36819: Network error: Client connection negotiation failed: client connection to 127.28.6.60:36819: connect: Connection refused (error 111)
W20260810 00:51:01.878722 30807 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 00:51:01.879081 30807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:51:01.879160 30807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:51:01.879273 30807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:51:01.886446 30807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:51:01.886523 30807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:51:01.886665 30807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:51:01.886759 30807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:51:01.886869 30807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:51:01.897261 30807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:45377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:42159,127.28.6.61:35219,127.28.6.60:36819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.60.keytab
--principal=kudu/127.28.6.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.60:36819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.6.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:51:01.898475 30807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:51:01.899904 30807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:51:01.911505 30822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:51:01.911553 30818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:51:01.911600 30817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:51:01.916285 30807 server_base.cc:1034] running on GCE node
I20260810 00:51:01.917040 30807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:51:01.918401 30807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:51:01.919620 30807 hybrid_clock.cc:648] HybridClock initialized: now 1786323061919557 us; error 33 us; skew 500 ppm
I20260810 00:51:01.919957 30807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:51:01 dist-test-slave-1nl6 krb5kdc[30570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323061, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:51:01.923506 30807 init.cc:377] Logged in from keytab as kudu/127.28.6.60@KRBTEST.COM (short username kudu)
I20260810 00:51:01.925123 30807 webserver.cc:460] Webserver started at http://127.28.6.60:43129/ using document root <none> and password file <none>
I20260810 00:51:01.925952 30807 fs_manager.cc:359] Metadata directory not provided
I20260810 00:51:01.926074 30807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:51:01.926435 30807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:51:01.930438 30807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-2/data/instance:
uuid: "bee47003a6424d8488ae48b018fe3acb"
format_stamp: "Formatted at 2026-08-10 00:51:01 on dist-test-slave-1nl6"
server_key: "acaed2ebfbdb855cf009c6c08f92802b"
server_key_iv: "60beb75655083b95637225696eed8237"
server_key_version: "encryptionkey@0"
I20260810 00:51:01.931416 30807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-2/wal/instance:
uuid: "bee47003a6424d8488ae48b018fe3acb"
format_stamp: "Formatted at 2026-08-10 00:51:01 on dist-test-slave-1nl6"
server_key: "acaed2ebfbdb855cf009c6c08f92802b"
server_key_iv: "60beb75655083b95637225696eed8237"
server_key_version: "encryptionkey@0"
I20260810 00:51:01.936841 30807 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 00:51:01.940114 30834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:51:01.941395 30807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 00:51:01.941541 30807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-2/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-2/wal
uuid: "bee47003a6424d8488ae48b018fe3acb"
format_stamp: "Formatted at 2026-08-10 00:51:01 on dist-test-slave-1nl6"
server_key: "acaed2ebfbdb855cf009c6c08f92802b"
server_key_iv: "60beb75655083b95637225696eed8237"
server_key_version: "encryptionkey@0"
I20260810 00:51:01.941735 30807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:51:01.967038 30807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:51:01.971335 30807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:51:01.971553 30807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:51:01.990087 30807 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.60:36819
I20260810 00:51:01.990106 30907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.60:36819 every 8 connection(s)
I20260810 00:51:01.991516 30807 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-2/data/info.pb
I20260810 00:51:01.995260 30908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:51:01.998879 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 30807
I20260810 00:51:01.999125 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786322687226232-28696-0/minicluster-data/master-2/wal/instance
I20260810 00:51:01.999564 28696 external_mini_cluster.cc:1428] Setting key 8684f8c1d1f1af76da23eceaa5b8aa01
I20260810 00:51:02.002667 30908 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.6.62" port: 42159 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 00:51:02 dist-test-slave-1nl6 krb5kdc[30570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.60@KRBTEST.COM
Aug 10 00:51:02 dist-test-slave-1nl6 krb5kdc[30570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323061, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.60@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:51:02.022804 30915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:02.006849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45536 (local address 127.28.6.62:42159)
0810 00:51:02.007511 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:02.007520 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:02.007981 (+ 461us) server_negotiation.cc:407] Connection header received
0810 00:51:02.008140 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:02.008144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:02.008231 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:02.008323 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:02.009660 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.010755 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.012534 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.012929 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.013062 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:02.016852 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:51:02.016857 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:51:02.016860 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:51:02.016893 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:51:02.019849 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:02.020600 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:02.020605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:02.020607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:02.020739 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:02.021144 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:02.021148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:02.021150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:02.021452 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:51:02.021612 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:02.022172 (+ 560us) server_negotiation.cc:299] Negotiation successful
0810 00:51:02.022395 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":339,"threads_started":1}
I20260810 00:51:02.023691 30916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:02.006958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:42159 (local address 127.0.0.1:45536)
0810 00:51:02.007815 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:02.007867 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 00:51:02.008023 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:51:02.008413 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:51:02.008419 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:51:02.009240 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:51:02.009492 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:51:02.009501 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.010921 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.010927 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:51:02.012382 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:51:02.012394 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.013161 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.013169 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:51:02.013321 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:02.013920 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:51:02.013941 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:51:02.016646 (+ 2705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:51:02.020002 (+ 3356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:51:02.020013 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:51:02.020024 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:51:02.020464 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:51:02.020858 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:51:02.020863 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:51:02.020867 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:51:02.021011 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:51:02.021590 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:51:02.021598 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:51:02.021805 (+ 207us) client_negotiation.cc:769] Sending connection context
0810 00:51:02.022034 (+ 229us) client_negotiation.cc:240] Negotiation successful
0810 00:51:02.022494 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"mutex_wait_us":3,"spinlock_wait_cycles":12672,"thread_start_us":450,"threads_started":1}
I20260810 00:51:02.023721 30911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:02.001504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43278 (local address 127.28.6.60:36819)
0810 00:51:02.002345 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:02.002360 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:02.002414 (+ 54us) server_negotiation.cc:407] Connection header received
0810 00:51:02.004146 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:02.004176 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:02.004688 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:02.005110 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:02.006824 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.008976 (+ 2152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.010656 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.011684 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.011865 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:02.016425 (+ 4560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:51:02.016434 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:51:02.016445 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:51:02.016484 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:51:02.019370 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:02.020106 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:02.020113 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:02.020115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:02.020290 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:02.020694 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:02.020700 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:02.020704 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:02.021206 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:51:02.021514 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:02.022015 (+ 501us) server_negotiation.cc:299] Negotiation successful
0810 00:51:02.022610 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":602,"spinlock_wait_cycles":20480,"thread_start_us":374,"threads_started":1}
I20260810 00:51:02.025648 30908 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.6.61" port: 35219 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 00:51:02 dist-test-slave-1nl6 krb5kdc[30570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323061, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.60@KRBTEST.COM for kudu/127.28.6.61@KRBTEST.COM
Aug 10 00:51:02 dist-test-slave-1nl6 krb5kdc[30570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:51:02.040230 30916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:02.026371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:35219 (local address 127.0.0.1:52596)
0810 00:51:02.026512 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:02.026528 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 00:51:02.026663 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:51:02.027255 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:51:02.027260 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:51:02.027755 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:51:02.027896 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:51:02.027905 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.029301 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.029307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:51:02.030558 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:51:02.030570 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.031511 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.031515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:51:02.031663 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:02.032200 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:51:02.032226 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:51:02.034429 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:51:02.037738 (+ 3309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:51:02.037745 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:51:02.037749 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:51:02.038108 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:51:02.038548 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:51:02.038553 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:51:02.038556 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:51:02.038701 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:51:02.039292 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:51:02.039297 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:51:02.039443 (+ 146us) client_negotiation.cc:769] Sending connection context
0810 00:51:02.039597 (+ 154us) client_negotiation.cc:240] Negotiation successful
0810 00:51:02.039858 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 00:51:02.040261 30802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:02.026641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52596 (local address 127.28.6.61:35219)
0810 00:51:02.026838 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:02.026845 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:02.026865 (+ 20us) server_negotiation.cc:407] Connection header received
0810 00:51:02.026981 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:02.027006 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:02.027095 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:02.027209 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:02.028055 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.029152 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.030706 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.031367 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.031497 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:02.034628 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:51:02.034635 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:51:02.034643 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:51:02.034697 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:51:02.037594 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:02.038251 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:02.038259 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:02.038262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:02.038408 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:02.038822 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:02.038827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:02.038830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:02.039153 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:51:02.039309 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:02.039697 (+ 388us) server_negotiation.cc:299] Negotiation successful
0810 00:51:02.039900 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 00:51:02.041186 30908 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.6.60" port: 36819 } has no permanent_uuid. Determining permanent_uuid...
I20260810 00:51:02.042482 30915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:02.027007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45550 (local address 127.28.6.62:42159)
0810 00:51:02.027199 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:02.027205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:02.027223 (+ 18us) server_negotiation.cc:407] Connection header received
0810 00:51:02.027392 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:02.027397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:02.027496 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:02.027599 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:02.029052 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.030148 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.031801 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.032500 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.032642 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:02.036909 (+ 4267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:51:02.036914 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:51:02.036917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:51:02.036945 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:51:02.039558 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:02.040206 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:02.040210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:02.040212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:02.040560 (+ 348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:02.040974 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:02.040980 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:02.040983 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:02.041296 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:51:02.041611 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:02.041905 (+ 294us) server_negotiation.cc:299] Negotiation successful
0810 00:51:02.042118 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Aug 10 00:51:02 dist-test-slave-1nl6 krb5kdc[30570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323061, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.60@KRBTEST.COM for kudu/127.28.6.60@KRBTEST.COM
Aug 10 00:51:02 dist-test-slave-1nl6 krb5kdc[30570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.61@KRBTEST.COM
I20260810 00:51:02.054759 30916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:02.041813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:36819 (local address 127.0.0.1:43280)
0810 00:51:02.042019 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:02.042036 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 00:51:02.042200 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:51:02.042483 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:51:02.042488 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:51:02.042844 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:51:02.043117 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:51:02.043131 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.044203 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.044209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:51:02.045245 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:51:02.045257 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.046251 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.046260 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:51:02.046484 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:02.047037 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:51:02.047067 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:51:02.049264 (+ 2197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:51:02.052396 (+ 3132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:51:02.052403 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:51:02.052407 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:51:02.052861 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:51:02.053229 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:51:02.053240 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:51:02.053244 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:51:02.053434 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:51:02.053956 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:51:02.053963 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:51:02.054137 (+ 174us) client_negotiation.cc:769] Sending connection context
0810 00:51:02.054289 (+ 152us) client_negotiation.cc:240] Negotiation successful
0810 00:51:02.054511 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
I20260810 00:51:02.055199 30911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:02.041846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43280 (local address 127.28.6.60:36819)
0810 00:51:02.041979 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:02.041985 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:02.042105 (+ 120us) server_negotiation.cc:407] Connection header received
0810 00:51:02.042304 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:02.042309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:02.042373 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:02.042470 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:02.043305 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.044060 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.045385 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.046084 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.046210 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:02.049444 (+ 3234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:51:02.049448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:51:02.049451 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:51:02.049490 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:51:02.052269 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:02.052985 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:02.052989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:02.052991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:02.053101 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:02.053558 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:02.053561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:02.053563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:02.053824 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:51:02.053934 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:02.054399 (+ 465us) server_negotiation.cc:299] Negotiation successful
0810 00:51:02.054714 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":41}
I20260810 00:51:02.055649 30683 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.60" port: 36819 } attempt: 4
I20260810 00:51:02.060864 30802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:02.044208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52604 (local address 127.28.6.61:35219)
0810 00:51:02.044373 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:02.044378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:02.044532 (+ 154us) server_negotiation.cc:407] Connection header received
0810 00:51:02.044690 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:02.044695 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:02.044755 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:02.044856 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:02.046310 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.047400 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.048867 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.049270 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.049405 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:02.054337 (+ 4932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:51:02.054345 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:51:02.054349 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:51:02.054384 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:51:02.057858 (+ 3474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:02.058659 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:02.058664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:02.058667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:02.058836 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:02.059312 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:02.059317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:02.059321 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:02.059664 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:51:02.059957 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:02.060265 (+ 308us) server_negotiation.cc:299] Negotiation successful
0810 00:51:02.060483 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 00:51:02.061317 30908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bee47003a6424d8488ae48b018fe3acb: Bootstrap starting.
Aug 10 00:51:02 dist-test-slave-1nl6 krb5kdc[30570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323060, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for kudu/127.28.6.60@KRBTEST.COM
I20260810 00:51:02.065596 30908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bee47003a6424d8488ae48b018fe3acb: Neither blocks nor log segments found. Creating new log.
I20260810 00:51:02.066107 28696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 00:51:02.067107 30908 log.cc:824] T 00000000000000000000000000000000 P bee47003a6424d8488ae48b018fe3acb: Log is configured to *not* fsync() on all Append() calls
I20260810 00:51:02.070495 30908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bee47003a6424d8488ae48b018fe3acb: No bootstrap required, opened a new log
I20260810 00:51:02.069301 30646 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4e478568994b463094671ad36417c3d3"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45550
I20260810 00:51:02.072789 30685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:02.056242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:36819 (local address 127.0.0.1:43284)
0810 00:51:02.056767 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:02.056788 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 00:51:02.056903 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:51:02.057295 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:51:02.057300 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:51:02.057805 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:51:02.058041 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:51:02.058054 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.059118 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.059123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:51:02.060624 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:51:02.060641 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.061941 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.061949 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:51:02.062159 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:02.062812 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:51:02.062835 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:51:02.065584 (+ 2749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:51:02.069494 (+ 3910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:51:02.069499 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:51:02.069504 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:51:02.070055 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:51:02.070563 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:51:02.070568 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:51:02.070572 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:51:02.070765 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:51:02.071661 (+ 896us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:51:02.071667 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:51:02.071858 (+ 191us) client_negotiation.cc:769] Sending connection context
0810 00:51:02.072031 (+ 173us) client_negotiation.cc:240] Negotiation successful
0810 00:51:02.072379 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403}
I20260810 00:51:02.073685 30911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:02.056576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43284 (local address 127.28.6.60:36819)
0810 00:51:02.056742 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:02.056748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:02.056865 (+ 117us) server_negotiation.cc:407] Connection header received
0810 00:51:02.057036 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:02.057043 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:02.057113 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:02.057212 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:02.058198 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.058977 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.060794 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.061758 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.062450 (+ 692us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:02.065823 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:51:02.065829 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:51:02.065833 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:51:02.065870 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:51:02.069352 (+ 3482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:02.070214 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:02.070219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:02.070222 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:02.070401 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:02.070990 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:02.071000 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:02.071005 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:02.071450 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:51:02.071588 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:02.073060 (+ 1472us) server_negotiation.cc:299] Negotiation successful
0810 00:51:02.073302 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 00:51:02.079336 30683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e478568994b463094671ad36417c3d3: Bootstrap starting.
I20260810 00:51:02.081295 30908 raft_consensus.cc:348] T 00000000000000000000000000000000 P bee47003a6424d8488ae48b018fe3acb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e478568994b463094671ad36417c3d3" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 42159 } } peers { permanent_uuid: "0839701233a94e9092be66da3197ddc4" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 35219 } } peers { permanent_uuid: "bee47003a6424d8488ae48b018fe3acb" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 36819 } }
I20260810 00:51:02.081710 30908 raft_consensus.cc:729] T 00000000000000000000000000000000 P bee47003a6424d8488ae48b018fe3acb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bee47003a6424d8488ae48b018fe3acb, State: Initialized, Role: FOLLOWER
I20260810 00:51:02.082198 30908 consensus_queue.cc:260] T 00000000000000000000000000000000 P bee47003a6424d8488ae48b018fe3acb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e478568994b463094671ad36417c3d3" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 42159 } } peers { permanent_uuid: "0839701233a94e9092be66da3197ddc4" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 35219 } } peers { permanent_uuid: "bee47003a6424d8488ae48b018fe3acb" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 36819 } }
I20260810 00:51:02.083146 30683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e478568994b463094671ad36417c3d3: Neither blocks nor log segments found. Creating new log.
I20260810 00:51:02.083207 30919 sys_catalog.cc:455] T 00000000000000000000000000000000 P bee47003a6424d8488ae48b018fe3acb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e478568994b463094671ad36417c3d3" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 42159 } } peers { permanent_uuid: "0839701233a94e9092be66da3197ddc4" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 35219 } } peers { permanent_uuid: "bee47003a6424d8488ae48b018fe3acb" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 36819 } } }
I20260810 00:51:02.083650 30919 sys_catalog.cc:458] T 00000000000000000000000000000000 P bee47003a6424d8488ae48b018fe3acb [sys.catalog]: This master's current role is: FOLLOWER
I20260810 00:51:02.084486 30908 sys_catalog.cc:564] T 00000000000000000000000000000000 P bee47003a6424d8488ae48b018fe3acb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:51:02.084546 30683 log.cc:824] T 00000000000000000000000000000000 P 4e478568994b463094671ad36417c3d3: Log is configured to *not* fsync() on all Append() calls
I20260810 00:51:02.088321 30683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e478568994b463094671ad36417c3d3: No bootstrap required, opened a new log
I20260810 00:51:02.096436 30683 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e478568994b463094671ad36417c3d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e478568994b463094671ad36417c3d3" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 42159 } } peers { permanent_uuid: "0839701233a94e9092be66da3197ddc4" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 35219 } } peers { permanent_uuid: "bee47003a6424d8488ae48b018fe3acb" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 36819 } }
I20260810 00:51:02.096771 30683 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e478568994b463094671ad36417c3d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e478568994b463094671ad36417c3d3, State: Initialized, Role: FOLLOWER
I20260810 00:51:02.097201 30683 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e478568994b463094671ad36417c3d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e478568994b463094671ad36417c3d3" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 42159 } } peers { permanent_uuid: "0839701233a94e9092be66da3197ddc4" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 35219 } } peers { permanent_uuid: "bee47003a6424d8488ae48b018fe3acb" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 36819 } }
I20260810 00:51:02.098163 30935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e478568994b463094671ad36417c3d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e478568994b463094671ad36417c3d3" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 42159 } } peers { permanent_uuid: "0839701233a94e9092be66da3197ddc4" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 35219 } } peers { permanent_uuid: "bee47003a6424d8488ae48b018fe3acb" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 36819 } } }
I20260810 00:51:02.098592 30935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e478568994b463094671ad36417c3d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 00:51:02.099385 30683 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e478568994b463094671ad36417c3d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:51:02.099448 30646 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e478568994b463094671ad36417c3d3 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 00:51:02.099620 30646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e478568994b463094671ad36417c3d3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:51:02.103749 30646 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e478568994b463094671ad36417c3d3 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e478568994b463094671ad36417c3d3" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 42159 } } peers { permanent_uuid: "0839701233a94e9092be66da3197ddc4" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 35219 } } peers { permanent_uuid: "bee47003a6424d8488ae48b018fe3acb" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 36819 } }
I20260810 00:51:02.107842 30874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4e478568994b463094671ad36417c3d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bee47003a6424d8488ae48b018fe3acb"
I20260810 00:51:02.108541 30874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bee47003a6424d8488ae48b018fe3acb [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:51:02.097693 30807 master_runner.cc:186] Error getting master registration for 127.28.6.62:42159: OK, instance_id { permanent_uuid: "4e478568994b463094671ad36417c3d3" instance_seqno: 1786323060917153 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 00:51:02.111397 30807 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 00:51:02.112851 30932 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bee47003a6424d8488ae48b018fe3acb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 00:51:02.112937 30932 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 00:51:02.113324 30874 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bee47003a6424d8488ae48b018fe3acb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e478568994b463094671ad36417c3d3 in term 1.
I20260810 00:51:02.114042 30610 leader_election.cc:302] T 00000000000000000000000000000000 P 4e478568994b463094671ad36417c3d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e478568994b463094671ad36417c3d3, bee47003a6424d8488ae48b018fe3acb; no voters:
I20260810 00:51:02.114097 30646 leader_election.cc:288] T 00000000000000000000000000000000 P 4e478568994b463094671ad36417c3d3 [CANDIDATE]: Term 1 election: Requested vote from peers 0839701233a94e9092be66da3197ddc4 (127.28.6.61:35219), bee47003a6424d8488ae48b018fe3acb (127.28.6.60:36819)
I20260810 00:51:02.114549 30935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e478568994b463094671ad36417c3d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:51:02.121470 30935 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e478568994b463094671ad36417c3d3 [term 1 LEADER]: Becoming Leader. State: Replica: 4e478568994b463094671ad36417c3d3, State: Running, Role: LEADER
I20260810 00:51:02.122061 30935 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e478568994b463094671ad36417c3d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e478568994b463094671ad36417c3d3" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 42159 } } peers { permanent_uuid: "0839701233a94e9092be66da3197ddc4" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 35219 } } peers { permanent_uuid: "bee47003a6424d8488ae48b018fe3acb" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 36819 } }
I20260810 00:51:02.129889 30938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e478568994b463094671ad36417c3d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e478568994b463094671ad36417c3d3. Latest consensus state: current_term: 1 leader_uuid: "4e478568994b463094671ad36417c3d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e478568994b463094671ad36417c3d3" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 42159 } } peers { permanent_uuid: "0839701233a94e9092be66da3197ddc4" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 35219 } } peers { permanent_uuid: "bee47003a6424d8488ae48b018fe3acb" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 36819 } } }
I20260810 00:51:02.130309 30938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e478568994b463094671ad36417c3d3 [sys.catalog]: This master's current role is: LEADER
I20260810 00:51:02.136410 30947 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:51:02.149444 30947 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260810 00:51:02.156216 30615 tablet.cc:2369] T 00000000000000000000000000000000 P 4e478568994b463094671ad36417c3d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:51:02.168900 30795 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.60" port: 36819 } attempt: 5
I20260810 00:51:02.172879 30874 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bee47003a6424d8488ae48b018fe3acb [term 1 FOLLOWER]: Refusing update from remote peer 4e478568994b463094671ad36417c3d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:51:02.174196 30935 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e478568994b463094671ad36417c3d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bee47003a6424d8488ae48b018fe3acb" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Aug 10 00:51:02 dist-test-slave-1nl6 krb5kdc[30570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323061, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.61@KRBTEST.COM for kudu/127.28.6.60@KRBTEST.COM
I20260810 00:51:02.191972 30800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:02.169445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:36819 (local address 127.0.0.1:43290)
0810 00:51:02.169774 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:02.169792 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 00:51:02.169900 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:51:02.170981 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:51:02.170989 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:51:02.171668 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:51:02.171947 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:51:02.171964 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.173990 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.173999 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:51:02.176007 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:51:02.176028 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.177667 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.177673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:51:02.177920 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:02.178781 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:51:02.178813 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:51:02.182606 (+ 3793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:51:02.188169 (+ 5563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:51:02.188177 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:51:02.188184 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:51:02.188880 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:51:02.189469 (+ 589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:51:02.189478 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:51:02.189484 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:51:02.189685 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:51:02.190579 (+ 894us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:51:02.190587 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:51:02.190826 (+ 239us) client_negotiation.cc:769] Sending connection context
0810 00:51:02.191032 (+ 206us) client_negotiation.cc:240] Negotiation successful
0810 00:51:02.191425 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221}
I20260810 00:51:02.193320 30911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:02.170112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43290 (local address 127.28.6.60:36819)
0810 00:51:02.170358 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:02.170366 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:02.170392 (+ 26us) server_negotiation.cc:407] Connection header received
0810 00:51:02.170473 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:02.170481 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:02.170609 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:02.170737 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:02.172197 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.173776 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.176229 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.177211 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.177373 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:02.182874 (+ 5501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:51:02.182885 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:51:02.182893 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:51:02.182979 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:51:02.187929 (+ 4950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:02.189054 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:02.189062 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:02.189069 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:02.189280 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:02.189837 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:02.189845 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:02.189851 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:02.190330 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:51:02.190502 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:02.192455 (+ 1953us) server_negotiation.cc:299] Negotiation successful
0810 00:51:02.192770 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 00:51:02.200842 30795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0839701233a94e9092be66da3197ddc4: Bootstrap starting.
I20260810 00:51:02.206099 30795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0839701233a94e9092be66da3197ddc4: Neither blocks nor log segments found. Creating new log.
I20260810 00:51:02.208336 30795 log.cc:824] T 00000000000000000000000000000000 P 0839701233a94e9092be66da3197ddc4: Log is configured to *not* fsync() on all Append() calls
W20260810 00:51:02.223706 30844 tablet.cc:2369] T 00000000000000000000000000000000 P bee47003a6424d8488ae48b018fe3acb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:51:02.229435 30795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0839701233a94e9092be66da3197ddc4: No bootstrap required, opened a new log
I20260810 00:51:02.243117 30919 sys_catalog.cc:455] T 00000000000000000000000000000000 P bee47003a6424d8488ae48b018fe3acb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e478568994b463094671ad36417c3d3. Latest consensus state: current_term: 1 leader_uuid: "4e478568994b463094671ad36417c3d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e478568994b463094671ad36417c3d3" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 42159 } } peers { permanent_uuid: "0839701233a94e9092be66da3197ddc4" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 35219 } } peers { permanent_uuid: "bee47003a6424d8488ae48b018fe3acb" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 36819 } } }
I20260810 00:51:02.243448 30919 sys_catalog.cc:458] T 00000000000000000000000000000000 P bee47003a6424d8488ae48b018fe3acb [sys.catalog]: This master's current role is: FOLLOWER
I20260810 00:51:02.246412 30958 mvcc.cc:205] Tried to move back new op lower bound from 7316779262645374976 to 7316779262458028032. Current Snapshot: MvccSnapshot[applied={T|T < 7316779262645374976}]
I20260810 00:51:02.249305 30915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:02.152055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45552 (local address 127.28.6.62:42159)
0810 00:51:02.152329 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:02.152340 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:02.167698 (+ 15358us) server_negotiation.cc:407] Connection header received
0810 00:51:02.168036 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:02.168045 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:02.168150 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:02.168261 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:02.210125 (+ 41864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.211707 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.219410 (+ 7703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.220442 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.220572 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:02.239649 (+ 19077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:51:02.239659 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:51:02.239667 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:51:02.239743 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:51:02.244540 (+ 4797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:02.245771 (+ 1231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:02.245781 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:02.245787 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:02.246012 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:02.246690 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:02.246698 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:02.246704 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:02.247265 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:51:02.247702 (+ 437us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:02.248437 (+ 735us) server_negotiation.cc:299] Negotiation successful
0810 00:51:02.248739 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 00:51:02.258684 30919 sys_catalog.cc:455] T 00000000000000000000000000000000 P bee47003a6424d8488ae48b018fe3acb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e478568994b463094671ad36417c3d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e478568994b463094671ad36417c3d3" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 42159 } } peers { permanent_uuid: "0839701233a94e9092be66da3197ddc4" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 35219 } } peers { permanent_uuid: "bee47003a6424d8488ae48b018fe3acb" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 36819 } } }
I20260810 00:51:02.258988 30919 sys_catalog.cc:458] T 00000000000000000000000000000000 P bee47003a6424d8488ae48b018fe3acb [sys.catalog]: This master's current role is: FOLLOWER
I20260810 00:51:02.261780 30938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e478568994b463094671ad36417c3d3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e478568994b463094671ad36417c3d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e478568994b463094671ad36417c3d3" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 42159 } } peers { permanent_uuid: "0839701233a94e9092be66da3197ddc4" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 35219 } } peers { permanent_uuid: "bee47003a6424d8488ae48b018fe3acb" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 36819 } } }
I20260810 00:51:02.262171 30938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e478568994b463094671ad36417c3d3 [sys.catalog]: This master's current role is: LEADER
I20260810 00:51:02.269368 30947 catalog_manager.cc:1347] Generated new cluster ID: 4ea54e645bf9412d9d808474e83ce59b
I20260810 00:51:02.269497 30947 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:51:02.278515 30911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:02.205732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43296 (local address 127.28.6.60:36819)
0810 00:51:02.207290 (+ 1558us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:02.207299 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:02.228057 (+ 20758us) server_negotiation.cc:407] Connection header received
0810 00:51:02.229594 (+ 1537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:02.229604 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:02.229722 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:02.229814 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:02.245216 (+ 15402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.246214 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.254851 (+ 8637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.255645 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.255777 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:02.269825 (+ 14048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:51:02.269833 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:51:02.269837 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:51:02.269881 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:51:02.273902 (+ 4021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:02.275298 (+ 1396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:02.275304 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:02.275308 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:02.275486 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:02.276209 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:02.276214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:02.276218 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:02.276565 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:51:02.276843 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:02.277789 (+ 946us) server_negotiation.cc:299] Negotiation successful
0810 00:51:02.278054 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1373,"thread_start_us":20322,"threads_started":1}
I20260810 00:51:02.311764 30795 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0839701233a94e9092be66da3197ddc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e478568994b463094671ad36417c3d3" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 42159 } } peers { permanent_uuid: "0839701233a94e9092be66da3197ddc4" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 35219 } } peers { permanent_uuid: "bee47003a6424d8488ae48b018fe3acb" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 36819 } }
I20260810 00:51:02.312282 30795 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0839701233a94e9092be66da3197ddc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0839701233a94e9092be66da3197ddc4, State: Initialized, Role: FOLLOWER
I20260810 00:51:02.312853 30795 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0839701233a94e9092be66da3197ddc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e478568994b463094671ad36417c3d3" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 42159 } } peers { permanent_uuid: "0839701233a94e9092be66da3197ddc4" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 35219 } } peers { permanent_uuid: "bee47003a6424d8488ae48b018fe3acb" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 36819 } }
I20260810 00:51:02.314201 30984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0839701233a94e9092be66da3197ddc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e478568994b463094671ad36417c3d3" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 42159 } } peers { permanent_uuid: "0839701233a94e9092be66da3197ddc4" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 35219 } } peers { permanent_uuid: "bee47003a6424d8488ae48b018fe3acb" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 36819 } } }
I20260810 00:51:02.314880 30984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0839701233a94e9092be66da3197ddc4 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 00:51:02.326097 30795 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0839701233a94e9092be66da3197ddc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:51:02.326187 30754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4e478568994b463094671ad36417c3d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0839701233a94e9092be66da3197ddc4"
I20260810 00:51:02.326682 30754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0839701233a94e9092be66da3197ddc4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:51:02.330298 30754 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0839701233a94e9092be66da3197ddc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e478568994b463094671ad36417c3d3 in term 1.
I20260810 00:51:02.331326 30752 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0839701233a94e9092be66da3197ddc4 [term 1 FOLLOWER]: Refusing update from remote peer 4e478568994b463094671ad36417c3d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:51:02.332223 30935 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e478568994b463094671ad36417c3d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0839701233a94e9092be66da3197ddc4" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 35219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:51:02.335844 30802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:02.167971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52616 (local address 127.28.6.61:35219)
0810 00:51:02.168147 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:02.168152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:02.216725 (+ 48573us) server_negotiation.cc:407] Connection header received
0810 00:51:02.218960 (+ 2235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:02.218967 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:02.219049 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:02.219122 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:02.236213 (+ 17091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.237373 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.259214 (+ 21841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.260411 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.260552 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:02.303466 (+ 42914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:51:02.303474 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:51:02.303480 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:51:02.303533 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:51:02.325643 (+ 22110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:02.327902 (+ 2259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:02.327911 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:02.327918 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:02.328121 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:02.328719 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:02.328727 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:02.328733 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:02.333725 (+ 4992us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:51:02.333987 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:02.335222 (+ 1235us) server_negotiation.cc:299] Negotiation successful
0810 00:51:02.335490 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 00:51:02.337567 30947 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:51:02.343784 30947 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:51:02.312387 30588 master_runner.cc:186] Error getting master registration for 127.28.6.61:35219: OK, instance_id { permanent_uuid: "0839701233a94e9092be66da3197ddc4" instance_seqno: 1786323061438759 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 00:51:02.360965 30588 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 00:51:02.367228 30990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0839701233a94e9092be66da3197ddc4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e478568994b463094671ad36417c3d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e478568994b463094671ad36417c3d3" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 42159 } } peers { permanent_uuid: "0839701233a94e9092be66da3197ddc4" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 35219 } } peers { permanent_uuid: "bee47003a6424d8488ae48b018fe3acb" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 36819 } } }
I20260810 00:51:02.367724 30990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0839701233a94e9092be66da3197ddc4 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 00:51:02.366665 30984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0839701233a94e9092be66da3197ddc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e478568994b463094671ad36417c3d3. Latest consensus state: current_term: 1 leader_uuid: "4e478568994b463094671ad36417c3d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e478568994b463094671ad36417c3d3" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 42159 } } peers { permanent_uuid: "0839701233a94e9092be66da3197ddc4" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 35219 } } peers { permanent_uuid: "bee47003a6424d8488ae48b018fe3acb" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 36819 } } }
I20260810 00:51:02.367919 30984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0839701233a94e9092be66da3197ddc4 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 00:51:02.371109 30938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e478568994b463094671ad36417c3d3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e478568994b463094671ad36417c3d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e478568994b463094671ad36417c3d3" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 42159 } } peers { permanent_uuid: "0839701233a94e9092be66da3197ddc4" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 35219 } } peers { permanent_uuid: "bee47003a6424d8488ae48b018fe3acb" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 36819 } } }
I20260810 00:51:02.371402 30938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e478568994b463094671ad36417c3d3 [sys.catalog]: This master's current role is: LEADER
I20260810 00:51:02.374594 30947 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e478568994b463094671ad36417c3d3: Generated new TSK 0
I20260810 00:51:02.375613 30947 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 00:51:02.425909 31010 catalog_manager.cc:1259] Loaded cluster ID: 4ea54e645bf9412d9d808474e83ce59b
I20260810 00:51:02.426015 31010 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0839701233a94e9092be66da3197ddc4: loading cluster ID for follower catalog manager: success
I20260810 00:51:02.430033 31010 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0839701233a94e9092be66da3197ddc4: acquiring CA information for follower catalog manager: success
I20260810 00:51:02.432519 31010 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0839701233a94e9092be66da3197ddc4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 00:51:02.499735 28696 mini_kdc.cc:348] Time spent kdestroy: real 0.113s user 0.004s sys 0.025s
W20260810 00:51:02.579171 30915 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:51:02.558249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45566 (local address 127.28.6.62:42159)
0810 00:51:02.558477 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:02.558485 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:02.567446 (+ 8961us) server_negotiation.cc:407] Connection header received
0810 00:51:02.572940 (+ 5494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:02.572947 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:02.573045 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:02.573160 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:02.578859 (+ 5699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45566: BlockingRecv error: recv got EOF from 127.0.0.1:45566 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260810 00:51:02.586031 30802 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:51:02.573860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52622 (local address 127.28.6.61:35219)
0810 00:51:02.574132 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:02.574171 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:02.580651 (+ 6480us) server_negotiation.cc:407] Connection header received
0810 00:51:02.584634 (+ 3983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:02.584641 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:02.584736 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:02.584809 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:02.585792 (+ 983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52622: BlockingRecv error: recv got EOF from 127.0.0.1:52622 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260810 00:51:02.591277 30911 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:51:02.588799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43300 (local address 127.28.6.60:36819)
0810 00:51:02.588986 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:02.588993 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:02.589464 (+ 471us) server_negotiation.cc:407] Connection header received
0810 00:51:02.589736 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:02.589741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:02.589832 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:02.589947 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:02.591041 (+ 1094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43300: BlockingRecv error: recv got EOF from 127.0.0.1:43300 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 00:51:02.644556 30802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:02.625907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52624 (local address 127.28.6.61:35219)
0810 00:51:02.626109 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:02.626116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:02.636340 (+ 10224us) server_negotiation.cc:407] Connection header received
0810 00:51:02.636900 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:02.636906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:02.637157 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:02.637298 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:51:02.638043 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.639202 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.641296 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.642020 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.642120 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:02.642961 (+ 841us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:51:02.643662 (+ 701us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:51:02.643800 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:02.644065 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 00:51:02.644182 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260810 00:51:02.691016 30911 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:51:02.677375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43308 (local address 127.28.6.60:36819)
0810 00:51:02.677583 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:02.677592 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:02.688741 (+ 11149us) server_negotiation.cc:407] Connection header received
0810 00:51:02.689319 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:02.689325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:02.689465 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:02.689551 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:02.690701 (+ 1150us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43308: BlockingRecv error: recv got EOF from 127.0.0.1:43308 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 00:51:02.744117 30802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:02.727850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52628 (local address 127.28.6.61:35219)
0810 00:51:02.728051 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:02.728060 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:02.736992 (+ 8932us) server_negotiation.cc:407] Connection header received
0810 00:51:02.737568 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:02.737574 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:02.737698 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:02.737772 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:51:02.738517 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.739674 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.741714 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.742420 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.742521 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:02.743189 (+ 668us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:51:02.743380 (+ 191us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:51:02.743447 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:02.743691 (+ 244us) server_negotiation.cc:299] Negotiation successful
0810 00:51:02.743791 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260810 00:51:02.792555 30911 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:51:02.780815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43316 (local address 127.28.6.60:36819)
0810 00:51:02.781038 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:02.781046 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:02.790187 (+ 9141us) server_negotiation.cc:407] Connection header received
0810 00:51:02.790776 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:02.790782 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:02.790884 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:02.791015 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:02.792299 (+ 1284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43316: BlockingRecv error: recv got EOF from 127.0.0.1:43316 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 00:51:02.831214 30802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:02.823142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52642 (local address 127.28.6.61:35219)
0810 00:51:02.823362 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:02.823369 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:02.823950 (+ 581us) server_negotiation.cc:407] Connection header received
0810 00:51:02.824172 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:02.824179 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:02.824318 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:02.824429 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:51:02.824937 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.826099 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.828212 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.828904 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.829065 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:02.829621 (+ 556us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:51:02.829776 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:51:02.829883 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:02.830127 (+ 244us) server_negotiation.cc:299] Negotiation successful
0810 00:51:02.830234 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260810 00:51:02.845202 30911 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:51:02.838992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43322 (local address 127.28.6.60:36819)
0810 00:51:02.839180 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:02.839189 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:02.843533 (+ 4344us) server_negotiation.cc:407] Connection header received
0810 00:51:02.843738 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:02.843747 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:02.843858 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:02.843978 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:02.844880 (+ 902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43322: BlockingRecv error: recv got EOF from 127.0.0.1:43322 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 00:51:02.859792 30802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:02.852491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52656 (local address 127.28.6.61:35219)
0810 00:51:02.852737 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:02.852743 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:02.853323 (+ 580us) server_negotiation.cc:407] Connection header received
0810 00:51:02.853535 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:02.853543 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:02.853643 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:02.853772 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:51:02.854258 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.855534 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.857540 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.858306 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.858451 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:02.858949 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:51:02.859091 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:51:02.859196 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:02.859392 (+ 196us) server_negotiation.cc:299] Negotiation successful
0810 00:51:02.859477 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
W20260810 00:51:02.879875 30911 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:51:02.877444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43336 (local address 127.28.6.60:36819)
0810 00:51:02.877692 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:02.877698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:02.877883 (+ 185us) server_negotiation.cc:407] Connection header received
0810 00:51:02.878236 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:02.878242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:02.878341 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:02.878425 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:02.879630 (+ 1205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43336: BlockingRecv error: recv got EOF from 127.0.0.1:43336 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
I20260810 00:51:02.903872 30802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:02.896858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52660 (local address 127.28.6.61:35219)
0810 00:51:02.897044 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:02.897050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:02.897559 (+ 509us) server_negotiation.cc:407] Connection header received
0810 00:51:02.897768 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:02.897774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:02.897877 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:02.898033 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:51:02.898506 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.899717 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.901621 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.902313 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.902442 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:02.902964 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:51:02.903107 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:51:02.903222 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:02.903397 (+ 175us) server_negotiation.cc:299] Negotiation successful
0810 00:51:02.903485 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260810 00:51:02.919440 30911 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:51:02.917067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43348 (local address 127.28.6.60:36819)
0810 00:51:02.917254 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:02.917260 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:02.917848 (+ 588us) server_negotiation.cc:407] Connection header received
0810 00:51:02.918028 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:02.918034 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:02.918142 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:02.918266 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:02.919172 (+ 906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43348: BlockingRecv error: recv got EOF from 127.0.0.1:43348 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 00:51:02.953751 30802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:02.948326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52676 (local address 127.28.6.61:35219)
0810 00:51:02.948527 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:02.948533 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:02.948992 (+ 459us) server_negotiation.cc:407] Connection header received
0810 00:51:02.949220 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:02.949236 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:02.949303 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:02.949406 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:51:02.949855 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.950667 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.952073 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.952518 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:02.952650 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:02.953068 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:51:02.953172 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:51:02.953284 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:02.953412 (+ 128us) server_negotiation.cc:299] Negotiation successful
0810 00:51:02.953497 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260810 00:51:02.961704 30911 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:51:02.959384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43354 (local address 127.28.6.60:36819)
0810 00:51:02.959580 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:02.959587 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:02.960095 (+ 508us) server_negotiation.cc:407] Connection header received
0810 00:51:02.960289 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:02.960293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:02.960363 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:02.960443 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:02.961458 (+ 1015us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43354: BlockingRecv error: recv got EOF from 127.0.0.1:43354 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 00:51:03.003652 30802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:02.997733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52690 (local address 127.28.6.61:35219)
0810 00:51:02.998018 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:02.998026 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:02.998342 (+ 316us) server_negotiation.cc:407] Connection header received
0810 00:51:02.998519 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:02.998523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:02.998603 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:02.998684 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:51:02.999134 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:02.999940 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:03.001908 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:03.002330 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:03.002423 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:03.002954 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:51:03.003058 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:51:03.003127 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:03.003286 (+ 159us) server_negotiation.cc:299] Negotiation successful
0810 00:51:03.003349 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260810 00:51:03.011549 30911 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:51:03.009566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43366 (local address 127.28.6.60:36819)
0810 00:51:03.009788 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:03.009794 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:03.010071 (+ 277us) server_negotiation.cc:407] Connection header received
0810 00:51:03.010245 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:03.010248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:03.010317 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:03.010403 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:03.011297 (+ 894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43366: BlockingRecv error: recv got EOF from 127.0.0.1:43366 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 00:51:03.083627 30802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:03.078707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52702 (local address 127.28.6.61:35219)
0810 00:51:03.078889 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:03.078956 (+ 67us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:03.079318 (+ 362us) server_negotiation.cc:407] Connection header received
0810 00:51:03.079553 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:03.079559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:03.079668 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:03.079781 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:51:03.080188 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:03.080936 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:03.082205 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:03.082627 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:03.082718 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:03.083083 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:51:03.083197 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:51:03.083266 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:03.083367 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 00:51:03.083427 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260810 00:51:03.090658 30911 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:51:03.089001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43380 (local address 127.28.6.60:36819)
0810 00:51:03.089182 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:03.089186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:03.089459 (+ 273us) server_negotiation.cc:407] Connection header received
0810 00:51:03.089588 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:03.089592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:03.089658 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:03.089738 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:03.090491 (+ 753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43380: BlockingRecv error: recv got EOF from 127.0.0.1:43380 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 00:51:03.114794 30932 catalog_manager.cc:1259] Loaded cluster ID: 4ea54e645bf9412d9d808474e83ce59b
I20260810 00:51:03.114889 30932 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bee47003a6424d8488ae48b018fe3acb: loading cluster ID for follower catalog manager: success
I20260810 00:51:03.117522 30932 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bee47003a6424d8488ae48b018fe3acb: acquiring CA information for follower catalog manager: success
I20260810 00:51:03.119323 30932 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bee47003a6424d8488ae48b018fe3acb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 00:51:03.229755 30802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:03.224606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52714 (local address 127.28.6.61:35219)
0810 00:51:03.224807 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:03.224814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:03.225284 (+ 470us) server_negotiation.cc:407] Connection header received
0810 00:51:03.225465 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:03.225469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:03.225547 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:03.225625 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:51:03.226022 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:03.226848 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:03.228274 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:03.228683 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:03.228776 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:03.229107 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:51:03.229205 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:51:03.229269 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:03.229395 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 00:51:03.229454 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 00:51:03.240300 30911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:03.234991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43382 (local address 127.28.6.60:36819)
0810 00:51:03.235248 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:03.235254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:03.235478 (+ 224us) server_negotiation.cc:407] Connection header received
0810 00:51:03.235603 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:03.235606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:03.235761 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:03.235856 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:51:03.236201 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:03.236913 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:03.238223 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:03.238617 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:03.238706 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:03.239244 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:51:03.239738 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:51:03.239848 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:03.239917 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 00:51:03.239977 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 00:51:03.252887 31163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:03.246354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45580 (local address 127.28.6.62:42159)
0810 00:51:03.247337 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:03.247343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:03.247366 (+ 23us) server_negotiation.cc:407] Connection header received
0810 00:51:03.247479 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:03.247482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:03.247632 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:03.247804 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:51:03.248441 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:03.249134 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:03.250470 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:03.250883 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:03.251064 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:03.251763 (+ 699us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:51:03.252206 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:51:03.252325 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:03.252494 (+ 169us) server_negotiation.cc:299] Negotiation successful
0810 00:51:03.252572 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":834,"thread_start_us":902,"threads_started":1}
I20260810 00:51:03.255445 30802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:03.247183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52728 (local address 127.28.6.61:35219)
0810 00:51:03.247365 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:03.247370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:03.247881 (+ 511us) server_negotiation.cc:407] Connection header received
0810 00:51:03.248212 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:03.248216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:03.248319 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:03.248391 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:51:03.249330 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:03.250748 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:03.253390 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:03.254002 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:03.254149 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:03.254708 (+ 559us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:51:03.254846 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:51:03.254980 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:03.255191 (+ 211us) server_negotiation.cc:299] Negotiation successful
0810 00:51:03.255265 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 00:51:03.258445 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 30588
I20260810 00:51:03.258718 30911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:03.248196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43396 (local address 127.28.6.60:36819)
0810 00:51:03.248370 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:03.248374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:03.251721 (+ 3347us) server_negotiation.cc:407] Connection header received
0810 00:51:03.251917 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:03.251928 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:03.251986 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:03.252088 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:51:03.252821 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:03.253855 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:03.256651 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:03.257318 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:03.257481 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:03.258112 (+ 631us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:51:03.258196 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:51:03.258271 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:03.258426 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 00:51:03.258481 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 00:51:03.259815 30673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:51:03.371693 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 30588
W20260810 00:51:03.400574 30716 connection.cc:535] server connection from 127.0.0.1:52580 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 00:51:03.401033 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 30693
I20260810 00:51:03.402172 30784 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:51:03.509357 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 30693
I20260810 00:51:03.540603 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 30807
I20260810 00:51:03.541779 30902 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:51:03.648046 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 30807
2026-08-10T00:51:03Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4827 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 00:51:03 dist-test-slave-1nl6 krb5kdc[31178](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 00:51:03 dist-test-slave-1nl6 krb5kdc[31178](info): set up 2 sockets
Aug 10 00:51:03 dist-test-slave-1nl6 krb5kdc[31178](info): commencing operation
krb5kdc: starting...
W20260810 00:51:04.972719 28696 mini_kdc.cc:121] Time spent starting KDC: real 1.273s user 0.003s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/krb5kdc/test-user.keytab.
Aug 10 00:51:05 dist-test-slave-1nl6 krb5kdc[31178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T00:51:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T00:51:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.6.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
Entry for principal kudu/127.28.6.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab.
I20260810 00:51:05.275738 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:34621
--webserver_interface=127.28.6.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:35759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:34621,127.28.6.61:39849,127.28.6.60:44249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 00:51:05.610724 31201 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 00:51:05.611053 31201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:51:05.611128 31201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:51:05.611209 31201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:51:05.618163 31201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:51:05.618252 31201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:51:05.618311 31201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:51:05.618363 31201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:51:05.618418 31201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:51:05.627817 31201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:35759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:34621,127.28.6.61:39849,127.28.6.60:44249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.62.keytab
--principal=kudu/127.28.6.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.62:34621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.6.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:51:05.629042 31201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:51:05.630313 31201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:51:05.641922 31216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:51:05.641883 31220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:51:05.641980 31217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:51:05.643061 31201 server_base.cc:1034] running on GCE node
I20260810 00:51:05.643796 31201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:51:05.645092 31201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:51:05.646318 31201 hybrid_clock.cc:648] HybridClock initialized: now 1786323065646265 us; error 29 us; skew 500 ppm
I20260810 00:51:05.646603 31201 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:51:05 dist-test-slave-1nl6 krb5kdc[31178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323065, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:51:05.649730 31201 init.cc:377] Logged in from keytab as kudu/127.28.6.62@KRBTEST.COM (short username kudu)
I20260810 00:51:05.651333 31201 webserver.cc:460] Webserver started at http://127.28.6.62:45799/ using document root <none> and password file <none>
I20260810 00:51:05.651999 31201 fs_manager.cc:359] Metadata directory not provided
I20260810 00:51:05.652117 31201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:51:05.652474 31201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:51:05.655435 31201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-0/data/instance:
uuid: "adc0c8fd4d3e43a9abdf2a1df839ed2c"
format_stamp: "Formatted at 2026-08-10 00:51:05 on dist-test-slave-1nl6"
server_key: "f162a4e1eb8d9bab3417e8aa6e257566"
server_key_iv: "9c03e30fe4af700834e774c5f82c0f17"
server_key_version: "encryptionkey@0"
I20260810 00:51:05.656230 31201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-0/wal/instance:
uuid: "adc0c8fd4d3e43a9abdf2a1df839ed2c"
format_stamp: "Formatted at 2026-08-10 00:51:05 on dist-test-slave-1nl6"
server_key: "f162a4e1eb8d9bab3417e8aa6e257566"
server_key_iv: "9c03e30fe4af700834e774c5f82c0f17"
server_key_version: "encryptionkey@0"
I20260810 00:51:05.661128 31201 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260810 00:51:05.664601 31229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:51:05.665845 31201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 00:51:05.666029 31201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-0/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-0/wal
uuid: "adc0c8fd4d3e43a9abdf2a1df839ed2c"
format_stamp: "Formatted at 2026-08-10 00:51:05 on dist-test-slave-1nl6"
server_key: "f162a4e1eb8d9bab3417e8aa6e257566"
server_key_iv: "9c03e30fe4af700834e774c5f82c0f17"
server_key_version: "encryptionkey@0"
I20260810 00:51:05.666239 31201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:51:05.679855 31201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:51:05.683593 31201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:51:05.683866 31201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:51:05.703368 31201 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.62:34621
I20260810 00:51:05.703392 31306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.62:34621 every 8 connection(s)
I20260810 00:51:05.704752 31201 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-0/data/info.pb
I20260810 00:51:05.708568 31307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:51:05.714800 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 31201
I20260810 00:51:05.715051 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-0/wal/instance
I20260810 00:51:05.715667 28696 external_mini_cluster.cc:1428] Setting key db488ecbc1a7b1811e3dc280440f5f4c
I20260810 00:51:05.715592 31307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34621 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 00:51:05 dist-test-slave-1nl6 krb5kdc[31178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323065, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:51:05.741319 31310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:05.719304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:34621 (local address 127.0.0.1:38996)
0810 00:51:05.720218 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:05.720268 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 00:51:05.722984 (+ 2716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:51:05.723793 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:51:05.723803 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:51:05.724608 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:51:05.725120 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:51:05.725148 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:51:05.726679 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:51:05.726687 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:51:05.728566 (+ 1879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:51:05.728585 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:51:05.729493 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:51:05.729503 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:51:05.729727 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:05.730435 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:51:05.730475 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:51:05.732980 (+ 2505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:51:05.737446 (+ 4466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:51:05.737457 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:51:05.737466 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:51:05.737868 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:51:05.738339 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:51:05.738342 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:51:05.738344 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:51:05.738492 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:51:05.739552 (+ 1060us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:51:05.739558 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:51:05.739760 (+ 202us) client_negotiation.cc:769] Sending connection context
0810 00:51:05.739968 (+ 208us) client_negotiation.cc:240] Negotiation successful
0810 00:51:05.740338 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":360,"threads_started":1}
I20260810 00:51:05.741381 31311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:05.719304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38996 (local address 127.28.6.62:34621)
0810 00:51:05.721147 (+ 1843us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:05.721153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:05.721196 (+ 43us) server_negotiation.cc:407] Connection header received
0810 00:51:05.723337 (+ 2141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:05.723342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:05.723550 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:05.723709 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:05.725344 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:05.726506 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:05.728709 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:05.729241 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:05.729366 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:05.733209 (+ 3843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:51:05.733222 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:51:05.733240 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:51:05.733286 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:51:05.737307 (+ 4021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:05.738025 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:05.738037 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:05.738043 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:05.738228 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:05.738617 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:05.738625 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:05.738632 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:05.739396 (+ 764us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:51:05.739610 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:05.740168 (+ 558us) server_negotiation.cc:299] Negotiation successful
0810 00:51:05.740538 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":1653,"thread_start_us":1425,"threads_started":1}
I20260810 00:51:05.744328 31307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.6.61" port: 39849 } has no permanent_uuid. Determining permanent_uuid...
I20260810 00:51:05.745370 31310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:05.744991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:39849 (local address 127.0.0.1:52094)
0810 00:51:05.745159 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:05.745273 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.61:39849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260810 00:51:05.745647 31232 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.6.61:39849: connect: Connection refused (error 111)
I20260810 00:51:05.747255 31310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:05.746844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:39849 (local address 127.0.0.1:52104)
0810 00:51:05.747050 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:05.747115 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.61:39849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260810 00:51:05.747503 31307 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.61:39849: Network error: Client connection negotiation failed: client connection to 127.28.6.61:39849: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.28.6.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.61.keytab.
Entry for principal kudu/127.28.6.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.61.keytab.
I20260810 00:51:05.790503 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.61:39849
--webserver_interface=127.28.6.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.61.keytab
--principal=kudu/127.28.6.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:35759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:34621,127.28.6.61:39849,127.28.6.60:44249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 00:51:05.796859 31307 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.61" port: 39849 } attempt: 1
I20260810 00:51:05.798167 31310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:05.797609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:39849 (local address 127.0.0.1:52116)
0810 00:51:05.797856 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:05.797971 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.61:39849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260810 00:51:05.799305 31310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:05.798821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:39849 (local address 127.0.0.1:52128)
0810 00:51:05.799023 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:05.799125 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.61:39849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260810 00:51:05.799512 31307 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.61:39849: Network error: Client connection negotiation failed: client connection to 127.28.6.61:39849: connect: Connection refused (error 111)
I20260810 00:51:05.867828 31307 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.61" port: 39849 } attempt: 2
I20260810 00:51:05.869176 31310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:05.868575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:39849 (local address 127.0.0.1:52144)
0810 00:51:05.868859 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:05.868985 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.61:39849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260810 00:51:05.870262 31310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:05.869856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:39849 (local address 127.0.0.1:52154)
0810 00:51:05.870012 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:05.870112 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.61:39849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 00:51:05.870460 31307 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.61:39849: Network error: Client connection negotiation failed: client connection to 127.28.6.61:39849: connect: Connection refused (error 111)
I20260810 00:51:05.961742 31307 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.61" port: 39849 } attempt: 3
I20260810 00:51:05.963114 31310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:05.962492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:39849 (local address 127.0.0.1:52162)
0810 00:51:05.962765 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:05.962888 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.61:39849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260810 00:51:05.964268 31310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:05.963776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:39849 (local address 127.0.0.1:52174)
0810 00:51:05.963961 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:05.964079 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.61:39849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260810 00:51:05.964486 31307 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.61:39849: Network error: Client connection negotiation failed: client connection to 127.28.6.61:39849: connect: Connection refused (error 111)
I20260810 00:51:06.107788 31307 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.61" port: 39849 } attempt: 4
I20260810 00:51:06.109087 31310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.108461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:39849 (local address 127.0.0.1:52188)
0810 00:51:06.108810 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:06.108921 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.61:39849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180}
I20260810 00:51:06.110172 31310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.109758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:39849 (local address 127.0.0.1:52204)
0810 00:51:06.109927 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:06.110021 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.61:39849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260810 00:51:06.110450 31307 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.61:39849: Network error: Client connection negotiation failed: client connection to 127.28.6.61:39849: connect: Connection refused (error 111)
W20260810 00:51:06.121615 31319 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 00:51:06.121933 31319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:51:06.122083 31319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:51:06.122208 31319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:51:06.129300 31319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:51:06.129376 31319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:51:06.129474 31319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:51:06.129552 31319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:51:06.129657 31319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:51:06.141294 31319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:35759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:34621,127.28.6.61:39849,127.28.6.60:44249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.61.keytab
--principal=kudu/127.28.6.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.61:39849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.6.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:51:06.143074 31319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:51:06.144956 31319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 00:51:06.156430 31332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:51:06.156466 31324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:51:06.156416 31328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:51:06.157130 31319 server_base.cc:1034] running on GCE node
I20260810 00:51:06.157899 31319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:51:06.159319 31319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:51:06.160552 31319 hybrid_clock.cc:648] HybridClock initialized: now 1786323066160468 us; error 56 us; skew 500 ppm
I20260810 00:51:06.160887 31319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:51:06 dist-test-slave-1nl6 krb5kdc[31178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323066, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:51:06.164057 31319 init.cc:377] Logged in from keytab as kudu/127.28.6.61@KRBTEST.COM (short username kudu)
I20260810 00:51:06.165393 31319 webserver.cc:460] Webserver started at http://127.28.6.61:46603/ using document root <none> and password file <none>
I20260810 00:51:06.166062 31319 fs_manager.cc:359] Metadata directory not provided
I20260810 00:51:06.166198 31319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:51:06.166529 31319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:51:06.169329 31319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-1/data/instance:
uuid: "44d1aa00c77946a8a0fbceed404bd1ec"
format_stamp: "Formatted at 2026-08-10 00:51:06 on dist-test-slave-1nl6"
server_key: "69cefa4e1dc140292025dbd35071e3ea"
server_key_iv: "77b81f7fb3311d875c50d7d41ec6c7fe"
server_key_version: "encryptionkey@0"
I20260810 00:51:06.170131 31319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-1/wal/instance:
uuid: "44d1aa00c77946a8a0fbceed404bd1ec"
format_stamp: "Formatted at 2026-08-10 00:51:06 on dist-test-slave-1nl6"
server_key: "69cefa4e1dc140292025dbd35071e3ea"
server_key_iv: "77b81f7fb3311d875c50d7d41ec6c7fe"
server_key_version: "encryptionkey@0"
I20260810 00:51:06.175161 31319 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 00:51:06.178263 31341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:51:06.179530 31319 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 00:51:06.179715 31319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-1/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-1/wal
uuid: "44d1aa00c77946a8a0fbceed404bd1ec"
format_stamp: "Formatted at 2026-08-10 00:51:06 on dist-test-slave-1nl6"
server_key: "69cefa4e1dc140292025dbd35071e3ea"
server_key_iv: "77b81f7fb3311d875c50d7d41ec6c7fe"
server_key_version: "encryptionkey@0"
I20260810 00:51:06.179940 31319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:51:06.191910 31319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:51:06.195513 31319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:51:06.195776 31319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:51:06.213673 31319 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.61:39849
I20260810 00:51:06.213691 31415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.61:39849 every 8 connection(s)
I20260810 00:51:06.214969 31319 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-1/data/info.pb
I20260810 00:51:06.217926 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 31319
I20260810 00:51:06.218101 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-1/wal/instance
I20260810 00:51:06.218544 31417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:51:06.218593 28696 external_mini_cluster.cc:1428] Setting key 43e4d06437eb6a030a0ff1f97a5bc9c0
I20260810 00:51:06.224678 31417 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34621 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 00:51:06 dist-test-slave-1nl6 krb5kdc[31178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323066, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.61@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:51:06.244745 31311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.227148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39010 (local address 127.28.6.62:34621)
0810 00:51:06.227347 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:06.227352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:06.228528 (+ 1176us) server_negotiation.cc:407] Connection header received
0810 00:51:06.230390 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:06.230396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:06.230490 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:06.230604 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:06.232103 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:06.232804 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:06.234290 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:06.234692 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:06.234825 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:06.238752 (+ 3927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:51:06.238757 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:51:06.238761 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:51:06.238791 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:51:06.241519 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:06.242493 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:06.242497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:06.242499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:06.242612 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:06.243101 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:06.243105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:06.243107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:06.243374 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:51:06.243482 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:06.244363 (+ 881us) server_negotiation.cc:299] Negotiation successful
0810 00:51:06.244527 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 00:51:06.246236 31419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.227641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:34621 (local address 127.0.0.1:39010)
0810 00:51:06.228363 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:06.228407 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 00:51:06.230118 (+ 1711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:51:06.230848 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:51:06.230855 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:51:06.231413 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:51:06.231926 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:51:06.231943 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:51:06.232972 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:51:06.232976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:51:06.234114 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:51:06.234127 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:51:06.234958 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:51:06.234968 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:51:06.235176 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:06.235846 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:51:06.235872 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:51:06.238539 (+ 2667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:51:06.241664 (+ 3125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:51:06.241676 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:51:06.241693 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:51:06.242372 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:51:06.242719 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:51:06.242725 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:51:06.242730 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:51:06.242984 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:51:06.243511 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:51:06.243521 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:51:06.243952 (+ 431us) client_negotiation.cc:769] Sending connection context
0810 00:51:06.244283 (+ 331us) client_negotiation.cc:240] Negotiation successful
0810 00:51:06.244794 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":304,"threads_started":1}
I20260810 00:51:06.248013 31417 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.6.61" port: 39849 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.6.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.6.60@KRBTEST.COM" created.
Aug 10 00:51:06 dist-test-slave-1nl6 krb5kdc[31178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323066, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.61@KRBTEST.COM for kudu/127.28.6.61@KRBTEST.COM
I20260810 00:51:06.263688 31419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.248865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:39849 (local address 127.0.0.1:52208)
0810 00:51:06.249063 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:06.249087 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 00:51:06.249207 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:51:06.250239 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:51:06.250242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:51:06.250621 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:51:06.250745 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:51:06.250752 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:51:06.252865 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:51:06.252869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:51:06.253791 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:51:06.253804 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:51:06.254563 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:51:06.254571 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:51:06.254777 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:06.255463 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:51:06.255487 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:51:06.257909 (+ 2422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:51:06.261027 (+ 3118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:51:06.261034 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:51:06.261038 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:51:06.261402 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:51:06.261790 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:51:06.261795 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:51:06.261798 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:51:06.261918 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:51:06.262478 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:51:06.262487 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:51:06.262634 (+ 147us) client_negotiation.cc:769] Sending connection context
0810 00:51:06.262790 (+ 156us) client_negotiation.cc:240] Negotiation successful
0810 00:51:06.263083 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 00:51:06.263741 31423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.249009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52208 (local address 127.28.6.61:39849)
0810 00:51:06.249724 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:06.249732 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:06.249761 (+ 29us) server_negotiation.cc:407] Connection header received
0810 00:51:06.249816 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:06.249826 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:06.250048 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:06.250243 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:06.250938 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:06.252721 (+ 1783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:06.253913 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:06.254426 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:06.254520 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:06.258114 (+ 3594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:51:06.258125 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:51:06.258138 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:51:06.258169 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:51:06.260879 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:06.261531 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:06.261540 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:06.261549 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:06.261656 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:06.262022 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:06.262029 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:06.262032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:06.262355 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:51:06.262570 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:06.262964 (+ 394us) server_negotiation.cc:299] Negotiation successful
0810 00:51:06.263170 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":393,"threads_started":1}
I20260810 00:51:06.266008 31417 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.6.60" port: 44249 } has no permanent_uuid. Determining permanent_uuid...
I20260810 00:51:06.267230 31419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.266757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:44249 (local address 127.0.0.1:43638)
0810 00:51:06.266964 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:06.267078 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:44249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260810 00:51:06.267585 31349 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.6.60:44249: connect: Connection refused (error 111)
I20260810 00:51:06.269320 31419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.268907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:44249 (local address 127.0.0.1:43654)
0810 00:51:06.269088 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:06.269179 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:44249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260810 00:51:06.269657 31417 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.60:44249: Network error: Client connection negotiation failed: client connection to 127.28.6.60:44249: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.6.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.60.keytab.
Entry for principal kudu/127.28.6.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.60.keytab.
I20260810 00:51:06.291886 28696 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksi0adg/build/asan/bin/kudu
/tmp/dist-test-tasksi0adg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.60:44249
--webserver_interface=127.28.6.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.60.keytab
--principal=kudu/127.28.6.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.6.20:35759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.6.62:34621,127.28.6.61:39849,127.28.6.60:44249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 00:51:06.319012 31417 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.60" port: 44249 } attempt: 1
I20260810 00:51:06.320207 31419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.319772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:44249 (local address 127.0.0.1:43656)
0810 00:51:06.319970 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:06.320057 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:44249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260810 00:51:06.321287 31419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.320930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:44249 (local address 127.0.0.1:43668)
0810 00:51:06.321090 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:06.321175 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:44249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 00:51:06.321496 31417 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.60:44249: Network error: Client connection negotiation failed: client connection to 127.28.6.60:44249: connect: Connection refused (error 111)
I20260810 00:51:06.389694 31417 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.60" port: 44249 } attempt: 2
I20260810 00:51:06.390779 31419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.390328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:44249 (local address 127.0.0.1:43676)
0810 00:51:06.390526 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:06.390630 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:44249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260810 00:51:06.391824 31419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.391501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:44249 (local address 127.0.0.1:43684)
0810 00:51:06.391647 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:06.391708 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:44249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 00:51:06.392035 31417 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.60:44249: Network error: Client connection negotiation failed: client connection to 127.28.6.60:44249: connect: Connection refused (error 111)
I20260810 00:51:06.409696 31307 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.61" port: 39849 } attempt: 5
Aug 10 00:51:06 dist-test-slave-1nl6 krb5kdc[31178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323065, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for kudu/127.28.6.61@KRBTEST.COM
I20260810 00:51:06.423426 31310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.410339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:39849 (local address 127.0.0.1:52222)
0810 00:51:06.410549 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:06.410570 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 00:51:06.410720 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:51:06.411131 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:51:06.411137 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:51:06.411646 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:51:06.411890 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:51:06.411904 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:51:06.412877 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:51:06.412883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:51:06.414288 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:51:06.414309 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:51:06.415086 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:51:06.415092 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:51:06.415298 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:06.415924 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:51:06.415947 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:51:06.418235 (+ 2288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:51:06.421108 (+ 2873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:51:06.421114 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:51:06.421118 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:51:06.421484 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:51:06.421854 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:51:06.421859 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:51:06.421862 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:51:06.421999 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:51:06.422518 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:51:06.422523 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:51:06.422668 (+ 145us) client_negotiation.cc:769] Sending connection context
0810 00:51:06.422809 (+ 141us) client_negotiation.cc:240] Negotiation successful
0810 00:51:06.423081 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260810 00:51:06.423441 31423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.410436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52222 (local address 127.28.6.61:39849)
0810 00:51:06.410584 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:06.410588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:06.410655 (+ 67us) server_negotiation.cc:407] Connection header received
0810 00:51:06.410837 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:06.410841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:06.410968 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:06.411058 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:06.412033 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:06.412712 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:06.414459 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:06.414883 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:06.415041 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:06.418435 (+ 3394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:51:06.418440 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:51:06.418443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:51:06.418470 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:51:06.420972 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:06.421610 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:06.421620 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:06.421623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:06.421733 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:06.422108 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:06.422113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:06.422117 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:06.422391 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:51:06.422537 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:06.422942 (+ 405us) server_negotiation.cc:299] Negotiation successful
0810 00:51:06.423152 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 00:51:06.424497 31307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.6.60" port: 44249 } has no permanent_uuid. Determining permanent_uuid...
I20260810 00:51:06.425587 31310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.425243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:44249 (local address 127.0.0.1:43690)
0810 00:51:06.425392 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:06.425459 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:44249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 00:51:06.426563 31310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.426227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:44249 (local address 127.0.0.1:43692)
0810 00:51:06.426382 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:06.426453 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:44249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260810 00:51:06.426765 31307 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.60:44249: Network error: Client connection negotiation failed: client connection to 127.28.6.60:44249: connect: Connection refused (error 111)
I20260810 00:51:06.477977 31307 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.60" port: 44249 } attempt: 1
I20260810 00:51:06.479152 31310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.478639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:44249 (local address 127.0.0.1:43694)
0810 00:51:06.478841 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:06.478988 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:44249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260810 00:51:06.480157 31310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.479814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:44249 (local address 127.0.0.1:43710)
0810 00:51:06.479970 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:06.480048 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:44249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 00:51:06.480420 31307 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.60:44249: Network error: Client connection negotiation failed: client connection to 127.28.6.60:44249: connect: Connection refused (error 111)
I20260810 00:51:06.483229 31417 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.60" port: 44249 } attempt: 3
I20260810 00:51:06.484200 31419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.483793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:44249 (local address 127.0.0.1:43726)
0810 00:51:06.483990 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:06.484064 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:44249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 00:51:06.485291 31419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.484886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:44249 (local address 127.0.0.1:43728)
0810 00:51:06.485105 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:06.485184 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:44249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260810 00:51:06.485498 31417 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.60:44249: Network error: Client connection negotiation failed: client connection to 127.28.6.60:44249: connect: Connection refused (error 111)
I20260810 00:51:06.548641 31307 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.60" port: 44249 } attempt: 2
I20260810 00:51:06.549811 31310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.549312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:44249 (local address 127.0.0.1:43738)
0810 00:51:06.549520 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:06.549630 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:44249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260810 00:51:06.550879 31310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.550542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:44249 (local address 127.0.0.1:43746)
0810 00:51:06.550716 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:06.550782 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:44249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 00:51:06.551115 31307 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.60:44249: Network error: Client connection negotiation failed: client connection to 127.28.6.60:44249: connect: Connection refused (error 111)
I20260810 00:51:06.628742 31417 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.60" port: 44249 } attempt: 4
I20260810 00:51:06.629896 31419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.629401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:44249 (local address 127.0.0.1:43752)
0810 00:51:06.629650 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:06.629734 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:44249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260810 00:51:06.631141 31419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.630642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:44249 (local address 127.0.0.1:43760)
0810 00:51:06.630872 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:06.630994 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:44249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260810 00:51:06.631479 31417 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.60:44249: Network error: Client connection negotiation failed: client connection to 127.28.6.60:44249: connect: Connection refused (error 111)
W20260810 00:51:06.638602 31427 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 00:51:06.638976 31427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 00:51:06.639097 31427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 00:51:06.639237 31427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 00:51:06.646360 31427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 00:51:06.646435 31427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 00:51:06.646559 31427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 00:51:06.646653 31427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 00:51:06.646757 31427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 00:51:06.656373 31427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.6.20:35759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.6.62:34621,127.28.6.61:39849,127.28.6.60:44249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/krb5kdc/kudu_127.28.6.60.keytab
--principal=kudu/127.28.6.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.6.60:44249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.6.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 23:54:02 UTC on ccycloud.quasar-sxogyf.root.comops.site
build id 4576868
ASAN enabled
I20260810 00:51:06.657342 31307 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.60" port: 44249 } attempt: 3
I20260810 00:51:06.657582 31427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 00:51:06.658389 31310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.657958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:44249 (local address 127.0.0.1:43774)
0810 00:51:06.658165 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:06.658245 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:44249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260810 00:51:06.659332 31427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 00:51:06.659516 31310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.659075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:44249 (local address 127.0.0.1:43776)
0810 00:51:06.659273 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:06.659363 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.6.60:44249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260810 00:51:06.659789 31307 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.6.60:44249: Network error: Client connection negotiation failed: client connection to 127.28.6.60:44249: connect: Connection refused (error 111)
W20260810 00:51:06.671831 31434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:51:06.671964 31436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 00:51:06.671941 31438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 00:51:06.672993 31427 server_base.cc:1034] running on GCE node
I20260810 00:51:06.673731 31427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 00:51:06.675110 31427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 00:51:06.676354 31427 hybrid_clock.cc:648] HybridClock initialized: now 1786323066676290 us; error 52 us; skew 500 ppm
I20260810 00:51:06.676667 31427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 00:51:06 dist-test-slave-1nl6 krb5kdc[31178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323066, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 00:51:06.683986 31427 init.cc:377] Logged in from keytab as kudu/127.28.6.60@KRBTEST.COM (short username kudu)
I20260810 00:51:06.685518 31427 webserver.cc:460] Webserver started at http://127.28.6.60:39649/ using document root <none> and password file <none>
I20260810 00:51:06.686255 31427 fs_manager.cc:359] Metadata directory not provided
I20260810 00:51:06.686389 31427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 00:51:06.686715 31427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 00:51:06.689634 31427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-2/data/instance:
uuid: "45e57f82ff1d42e0808bd13fb7d72e24"
format_stamp: "Formatted at 2026-08-10 00:51:06 on dist-test-slave-1nl6"
server_key: "62e47e3be83436d7047f06a30d95df14"
server_key_iv: "fecdedf41c938b38a4ce3fa7749b967a"
server_key_version: "encryptionkey@0"
I20260810 00:51:06.690423 31427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-2/wal/instance:
uuid: "45e57f82ff1d42e0808bd13fb7d72e24"
format_stamp: "Formatted at 2026-08-10 00:51:06 on dist-test-slave-1nl6"
server_key: "62e47e3be83436d7047f06a30d95df14"
server_key_iv: "fecdedf41c938b38a4ce3fa7749b967a"
server_key_version: "encryptionkey@0"
I20260810 00:51:06.695472 31427 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 00:51:06.698891 31449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 00:51:06.700599 31427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 00:51:06.700789 31427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-2/data,/tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-2/wal
uuid: "45e57f82ff1d42e0808bd13fb7d72e24"
format_stamp: "Formatted at 2026-08-10 00:51:06 on dist-test-slave-1nl6"
server_key: "62e47e3be83436d7047f06a30d95df14"
server_key_iv: "fecdedf41c938b38a4ce3fa7749b967a"
server_key_version: "encryptionkey@0"
I20260810 00:51:06.701059 31427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 00:51:06.715595 31427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 00:51:06.719551 31427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 00:51:06.719779 31427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 00:51:06.739117 31427 rpc_server.cc:307] RPC server started. Bound to: 127.28.6.60:44249
I20260810 00:51:06.739135 31523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.6.60:44249 every 8 connection(s)
I20260810 00:51:06.740494 31427 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-2/data/info.pb
I20260810 00:51:06.740814 28696 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksi0adg/build/asan/bin/kudu as pid 31427
I20260810 00:51:06.741006 28696 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksi0adg/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786322687226232-28696-0/minicluster-data/master-2/wal/instance
I20260810 00:51:06.741482 28696 external_mini_cluster.cc:1428] Setting key 48ce5411c21e1cfd2e552c8927bff53e
I20260810 00:51:06.745174 31525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 00:51:06.752110 31525 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34621 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 00:51:06 dist-test-slave-1nl6 krb5kdc[31178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.60@KRBTEST.COM
Aug 10 00:51:06 dist-test-slave-1nl6 krb5kdc[31178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323066, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.60@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
I20260810 00:51:06.765964 31531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.743421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43788 (local address 127.28.6.60:44249)
0810 00:51:06.744376 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:06.744387 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:06.744428 (+ 41us) server_negotiation.cc:407] Connection header received
0810 00:51:06.746182 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:06.746205 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:06.746560 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:06.746960 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:06.748421 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:06.749792 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:06.751087 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:06.751825 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:06.752002 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:06.757085 (+ 5083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:51:06.757102 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:51:06.757117 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:51:06.757168 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:51:06.760960 (+ 3792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:06.761924 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:06.761930 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:06.761936 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:06.762197 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:06.762633 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:06.762639 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:06.762642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:06.763429 (+ 787us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:51:06.763782 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:06.764073 (+ 291us) server_negotiation.cc:299] Negotiation successful
0810 00:51:06.764607 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":735,"thread_start_us":519,"threads_started":1}
I20260810 00:51:06.772289 31532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.755185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39026 (local address 127.28.6.62:34621)
0810 00:51:06.755715 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:06.755721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:06.756301 (+ 580us) server_negotiation.cc:407] Connection header received
0810 00:51:06.756453 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:06.756458 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:06.756540 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:06.756661 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:06.758262 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:06.759222 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:06.761255 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:06.761956 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:06.762059 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:06.765986 (+ 3927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:51:06.765991 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:51:06.765994 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:51:06.766046 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:51:06.769375 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:06.770183 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:06.770189 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:06.770191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:06.770357 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:06.770740 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:06.770746 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:06.770750 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:06.771188 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:51:06.771304 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:06.771819 (+ 515us) server_negotiation.cc:299] Negotiation successful
0810 00:51:06.771978 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":258,"threads_started":1}
I20260810 00:51:06.772455 31535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.755152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.62:34621 (local address 127.0.0.1:39026)
0810 00:51:06.756123 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:06.756172 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 00:51:06.756322 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:51:06.756737 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:51:06.756742 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:51:06.757689 (+ 947us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:51:06.758096 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:51:06.758112 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:51:06.759389 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:51:06.759394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:51:06.761111 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:51:06.761123 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:51:06.762159 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:51:06.762165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:51:06.762390 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:06.763008 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:51:06.763029 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:51:06.765780 (+ 2751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:51:06.769502 (+ 3722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:51:06.769513 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:51:06.769520 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:51:06.770056 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:51:06.770488 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:51:06.770492 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:51:06.770494 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:51:06.770624 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:51:06.771324 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:51:06.771334 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:51:06.771522 (+ 188us) client_negotiation.cc:769] Sending connection context
0810 00:51:06.771740 (+ 218us) client_negotiation.cc:240] Negotiation successful
0810 00:51:06.772048 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"thread_start_us":591,"threads_started":1}
I20260810 00:51:06.773802 31525 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.6.61" port: 39849 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 00:51:06 dist-test-slave-1nl6 krb5kdc[31178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.62@KRBTEST.COM
Aug 10 00:51:06 dist-test-slave-1nl6 krb5kdc[31178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323066, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.60@KRBTEST.COM for kudu/127.28.6.61@KRBTEST.COM
I20260810 00:51:06.785127 31541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.768701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39032 (local address 127.28.6.62:34621)
0810 00:51:06.769376 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:06.769380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:06.769399 (+ 19us) server_negotiation.cc:407] Connection header received
0810 00:51:06.769469 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:06.769473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:06.769543 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:06.769669 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:06.771131 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:06.772393 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:06.774124 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:06.774633 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:06.774755 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:06.779393 (+ 4638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:51:06.779399 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:51:06.779402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:51:06.779430 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:51:06.782127 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:06.782857 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:06.782863 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:06.782867 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:06.783035 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:06.783445 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:06.783450 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:06.783453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:06.783828 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:51:06.784122 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:06.784433 (+ 311us) server_negotiation.cc:299] Negotiation successful
0810 00:51:06.784642 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":351,"threads_started":1}
I20260810 00:51:06.787919 31535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.774620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.61:39849 (local address 127.0.0.1:52234)
0810 00:51:06.774772 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:06.774792 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 00:51:06.774880 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:51:06.775266 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:51:06.775269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:51:06.775581 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:51:06.775712 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:51:06.775720 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:51:06.777095 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:51:06.777098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:51:06.777951 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:51:06.777962 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:51:06.778722 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:51:06.778729 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:51:06.778954 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:06.779496 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:51:06.779513 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:51:06.781721 (+ 2208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:51:06.785086 (+ 3365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:51:06.785091 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:51:06.785095 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:51:06.785565 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:51:06.785949 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:51:06.785954 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:51:06.785958 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:51:06.786135 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:51:06.786858 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:51:06.786884 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:51:06.787098 (+ 214us) client_negotiation.cc:769] Sending connection context
0810 00:51:06.787282 (+ 184us) client_negotiation.cc:240] Negotiation successful
0810 00:51:06.787549 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260810 00:51:06.788036 31423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.774633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52234 (local address 127.28.6.61:39849)
0810 00:51:06.774847 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:06.774853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:06.774875 (+ 22us) server_negotiation.cc:407] Connection header received
0810 00:51:06.775046 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:06.775051 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:06.775131 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:06.775243 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:06.775842 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:06.776943 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:06.778097 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:06.778569 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:06.778703 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:06.781926 (+ 3223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:51:06.781930 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:51:06.781934 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:51:06.781963 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:51:06.784953 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:06.785682 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:06.785688 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:06.785691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:06.785838 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:06.786271 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:06.786275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:06.786277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:06.786630 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:51:06.786802 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:06.787420 (+ 618us) server_negotiation.cc:299] Negotiation successful
0810 00:51:06.787597 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260810 00:51:06.788935 31525 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.6.60" port: 44249 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 00:51:06 dist-test-slave-1nl6 krb5kdc[31178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.6.61@KRBTEST.COM
Aug 10 00:51:06 dist-test-slave-1nl6 krb5kdc[31178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323066, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.60@KRBTEST.COM for kudu/127.28.6.60@KRBTEST.COM
I20260810 00:51:06.803978 31542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.786687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52238 (local address 127.28.6.61:39849)
0810 00:51:06.787354 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:06.787361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:06.787383 (+ 22us) server_negotiation.cc:407] Connection header received
0810 00:51:06.787429 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:06.787433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:06.787505 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:06.787613 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:06.789448 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:06.790698 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:06.792419 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:06.793074 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:06.793210 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:06.797559 (+ 4349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:51:06.797564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:51:06.797567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:51:06.797594 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:51:06.800910 (+ 3316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:06.801765 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:06.801769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:06.801771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:06.801927 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:06.802376 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:06.802379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:06.802381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:06.802742 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:51:06.803030 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:06.803357 (+ 327us) server_negotiation.cc:299] Negotiation successful
0810 00:51:06.803549 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":312,"threads_started":1}
I20260810 00:51:06.803998 31535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.789776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:44249 (local address 127.0.0.1:43802)
0810 00:51:06.790018 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:06.790035 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 00:51:06.790167 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:51:06.790450 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:51:06.790455 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:51:06.790963 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:51:06.791101 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:51:06.791110 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:51:06.792419 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:51:06.792424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:51:06.793611 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:51:06.793623 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:51:06.794339 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:51:06.794345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:51:06.794516 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:06.795041 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:51:06.795056 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:51:06.797612 (+ 2556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:51:06.801265 (+ 3653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:51:06.801270 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:51:06.801273 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:51:06.801785 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:51:06.802170 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:51:06.802178 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:51:06.802181 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:51:06.802348 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:51:06.802973 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:51:06.802978 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:51:06.803120 (+ 142us) client_negotiation.cc:769] Sending connection context
0810 00:51:06.803276 (+ 156us) client_negotiation.cc:240] Negotiation successful
0810 00:51:06.803570 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260810 00:51:06.804347 31531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.789711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43802 (local address 127.28.6.60:44249)
0810 00:51:06.789880 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:06.789884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:06.790112 (+ 228us) server_negotiation.cc:407] Connection header received
0810 00:51:06.790270 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:06.790273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:06.790321 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:06.790390 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:06.791237 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:06.792295 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:06.793759 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:06.794170 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:06.794299 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:06.797787 (+ 3488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:51:06.797793 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:51:06.797798 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:51:06.797832 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:51:06.801126 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:06.801894 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:06.801899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:06.801901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:06.802050 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:06.802466 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:06.802473 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:06.802476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:06.802825 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:51:06.802992 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:06.803371 (+ 379us) server_negotiation.cc:299] Negotiation successful
0810 00:51:06.803773 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":148,"server-negotiator.queue_time_us":71}
I20260810 00:51:06.809129 28696 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 00:51:06.809973 31525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45e57f82ff1d42e0808bd13fb7d72e24: Bootstrap starting.
I20260810 00:51:06.813192 31525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45e57f82ff1d42e0808bd13fb7d72e24: Neither blocks nor log segments found. Creating new log.
I20260810 00:51:06.811352 31267 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "adc0c8fd4d3e43a9abdf2a1df839ed2c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39032
I20260810 00:51:06.814288 31525 log.cc:824] T 00000000000000000000000000000000 P 45e57f82ff1d42e0808bd13fb7d72e24: Log is configured to *not* fsync() on all Append() calls
I20260810 00:51:06.817266 31525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45e57f82ff1d42e0808bd13fb7d72e24: No bootstrap required, opened a new log
I20260810 00:51:06.824223 31525 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45e57f82ff1d42e0808bd13fb7d72e24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc0c8fd4d3e43a9abdf2a1df839ed2c" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34621 } } peers { permanent_uuid: "44d1aa00c77946a8a0fbceed404bd1ec" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 39849 } } peers { permanent_uuid: "45e57f82ff1d42e0808bd13fb7d72e24" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 44249 } }
I20260810 00:51:06.824549 31525 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45e57f82ff1d42e0808bd13fb7d72e24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45e57f82ff1d42e0808bd13fb7d72e24, State: Initialized, Role: FOLLOWER
I20260810 00:51:06.824968 31525 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45e57f82ff1d42e0808bd13fb7d72e24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc0c8fd4d3e43a9abdf2a1df839ed2c" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34621 } } peers { permanent_uuid: "44d1aa00c77946a8a0fbceed404bd1ec" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 39849 } } peers { permanent_uuid: "45e57f82ff1d42e0808bd13fb7d72e24" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 44249 } }
I20260810 00:51:06.825883 31544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45e57f82ff1d42e0808bd13fb7d72e24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc0c8fd4d3e43a9abdf2a1df839ed2c" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34621 } } peers { permanent_uuid: "44d1aa00c77946a8a0fbceed404bd1ec" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 39849 } } peers { permanent_uuid: "45e57f82ff1d42e0808bd13fb7d72e24" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 44249 } } }
I20260810 00:51:06.826282 31544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45e57f82ff1d42e0808bd13fb7d72e24 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 00:51:06.826758 31525 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45e57f82ff1d42e0808bd13fb7d72e24 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 00:51:06.835673 31559 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 45e57f82ff1d42e0808bd13fb7d72e24: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 00:51:06.835822 31559 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 00:51:06.836989 31307 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.60" port: 44249 } attempt: 4
Aug 10 00:51:06 dist-test-slave-1nl6 krb5kdc[31178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323065, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.62@KRBTEST.COM for kudu/127.28.6.60@KRBTEST.COM
I20260810 00:51:06.833911 31427 master_runner.cc:186] Error getting master registration for 127.28.6.62:34621: OK, instance_id { permanent_uuid: "adc0c8fd4d3e43a9abdf2a1df839ed2c" instance_seqno: 1786323065697698 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 00:51:06.845991 31427 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 00:51:06.853078 31310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.837538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:44249 (local address 127.0.0.1:43804)
0810 00:51:06.837774 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:06.837815 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 00:51:06.837948 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:51:06.838318 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:51:06.838323 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:51:06.838815 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:51:06.839074 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:51:06.839087 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:51:06.840164 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:51:06.840171 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:51:06.841638 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:51:06.841653 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:51:06.842415 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:51:06.842421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:51:06.842658 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:06.843401 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:51:06.843423 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:51:06.846112 (+ 2689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:51:06.850233 (+ 4121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:51:06.850240 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:51:06.850247 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:51:06.850757 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:51:06.851186 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:51:06.851191 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:51:06.851195 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:51:06.851362 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:51:06.852055 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:51:06.852062 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:51:06.852257 (+ 195us) client_negotiation.cc:769] Sending connection context
0810 00:51:06.852406 (+ 149us) client_negotiation.cc:240] Negotiation successful
0810 00:51:06.852683 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260810 00:51:06.853134 31531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.837605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43804 (local address 127.28.6.60:44249)
0810 00:51:06.837808 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:06.837814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:06.837871 (+ 57us) server_negotiation.cc:407] Connection header received
0810 00:51:06.838066 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:06.838072 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:06.838149 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:06.838251 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:06.839231 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:06.840022 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:06.841787 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:06.842254 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:06.842395 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:06.846317 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:51:06.846325 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:51:06.846329 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:51:06.846363 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:51:06.850077 (+ 3714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:06.850886 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:06.850892 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:06.850895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:06.851068 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:06.851483 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:06.851489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:06.851492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:06.851905 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:51:06.852078 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:06.852542 (+ 464us) server_negotiation.cc:299] Negotiation successful
0810 00:51:06.852765 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 00:51:06.858852 31307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adc0c8fd4d3e43a9abdf2a1df839ed2c: Bootstrap starting.
I20260810 00:51:06.862403 31307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adc0c8fd4d3e43a9abdf2a1df839ed2c: Neither blocks nor log segments found. Creating new log.
I20260810 00:51:06.863543 31307 log.cc:824] T 00000000000000000000000000000000 P adc0c8fd4d3e43a9abdf2a1df839ed2c: Log is configured to *not* fsync() on all Append() calls
I20260810 00:51:06.866380 31307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adc0c8fd4d3e43a9abdf2a1df839ed2c: No bootstrap required, opened a new log
I20260810 00:51:06.871754 31307 raft_consensus.cc:348] T 00000000000000000000000000000000 P adc0c8fd4d3e43a9abdf2a1df839ed2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc0c8fd4d3e43a9abdf2a1df839ed2c" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34621 } } peers { permanent_uuid: "44d1aa00c77946a8a0fbceed404bd1ec" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 39849 } } peers { permanent_uuid: "45e57f82ff1d42e0808bd13fb7d72e24" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 44249 } }
I20260810 00:51:06.872094 31307 raft_consensus.cc:729] T 00000000000000000000000000000000 P adc0c8fd4d3e43a9abdf2a1df839ed2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adc0c8fd4d3e43a9abdf2a1df839ed2c, State: Initialized, Role: FOLLOWER
I20260810 00:51:06.872536 31307 consensus_queue.cc:260] T 00000000000000000000000000000000 P adc0c8fd4d3e43a9abdf2a1df839ed2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc0c8fd4d3e43a9abdf2a1df839ed2c" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34621 } } peers { permanent_uuid: "44d1aa00c77946a8a0fbceed404bd1ec" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 39849 } } peers { permanent_uuid: "45e57f82ff1d42e0808bd13fb7d72e24" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 44249 } }
I20260810 00:51:06.873466 31563 sys_catalog.cc:455] T 00000000000000000000000000000000 P adc0c8fd4d3e43a9abdf2a1df839ed2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc0c8fd4d3e43a9abdf2a1df839ed2c" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34621 } } peers { permanent_uuid: "44d1aa00c77946a8a0fbceed404bd1ec" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 39849 } } peers { permanent_uuid: "45e57f82ff1d42e0808bd13fb7d72e24" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 44249 } } }
I20260810 00:51:06.873898 31563 sys_catalog.cc:458] T 00000000000000000000000000000000 P adc0c8fd4d3e43a9abdf2a1df839ed2c [sys.catalog]: This master's current role is: FOLLOWER
I20260810 00:51:06.874604 31307 sys_catalog.cc:564] T 00000000000000000000000000000000 P adc0c8fd4d3e43a9abdf2a1df839ed2c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:51:06.874766 31267 raft_consensus.cc:482] T 00000000000000000000000000000000 P adc0c8fd4d3e43a9abdf2a1df839ed2c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 00:51:06.875068 31267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adc0c8fd4d3e43a9abdf2a1df839ed2c [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:51:06.878738 31267 raft_consensus.cc:504] T 00000000000000000000000000000000 P adc0c8fd4d3e43a9abdf2a1df839ed2c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc0c8fd4d3e43a9abdf2a1df839ed2c" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34621 } } peers { permanent_uuid: "44d1aa00c77946a8a0fbceed404bd1ec" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 39849 } } peers { permanent_uuid: "45e57f82ff1d42e0808bd13fb7d72e24" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 44249 } }
I20260810 00:51:06.880668 31267 leader_election.cc:288] T 00000000000000000000000000000000 P adc0c8fd4d3e43a9abdf2a1df839ed2c [CANDIDATE]: Term 1 election: Requested vote from peers 44d1aa00c77946a8a0fbceed404bd1ec (127.28.6.61:39849), 45e57f82ff1d42e0808bd13fb7d72e24 (127.28.6.60:44249)
I20260810 00:51:06.888170 31489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "adc0c8fd4d3e43a9abdf2a1df839ed2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "45e57f82ff1d42e0808bd13fb7d72e24"
I20260810 00:51:06.888741 31489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45e57f82ff1d42e0808bd13fb7d72e24 [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:51:06.893138 31489 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 45e57f82ff1d42e0808bd13fb7d72e24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adc0c8fd4d3e43a9abdf2a1df839ed2c in term 1.
I20260810 00:51:06.894524 31235 leader_election.cc:302] T 00000000000000000000000000000000 P adc0c8fd4d3e43a9abdf2a1df839ed2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45e57f82ff1d42e0808bd13fb7d72e24, adc0c8fd4d3e43a9abdf2a1df839ed2c; no voters:
I20260810 00:51:06.895068 31563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adc0c8fd4d3e43a9abdf2a1df839ed2c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 00:51:06.898278 31563 raft_consensus.cc:686] T 00000000000000000000000000000000 P adc0c8fd4d3e43a9abdf2a1df839ed2c [term 1 LEADER]: Becoming Leader. State: Replica: adc0c8fd4d3e43a9abdf2a1df839ed2c, State: Running, Role: LEADER
I20260810 00:51:06.898947 31563 consensus_queue.cc:237] T 00000000000000000000000000000000 P adc0c8fd4d3e43a9abdf2a1df839ed2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc0c8fd4d3e43a9abdf2a1df839ed2c" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34621 } } peers { permanent_uuid: "44d1aa00c77946a8a0fbceed404bd1ec" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 39849 } } peers { permanent_uuid: "45e57f82ff1d42e0808bd13fb7d72e24" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 44249 } }
W20260810 00:51:06.917773 31576 catalog_manager.cc:1558] T 00000000000000000000000000000000 P adc0c8fd4d3e43a9abdf2a1df839ed2c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 00:51:06.917945 31576 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 00:51:06.923213 31581 sys_catalog.cc:455] T 00000000000000000000000000000000 P adc0c8fd4d3e43a9abdf2a1df839ed2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader adc0c8fd4d3e43a9abdf2a1df839ed2c. Latest consensus state: current_term: 1 leader_uuid: "adc0c8fd4d3e43a9abdf2a1df839ed2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc0c8fd4d3e43a9abdf2a1df839ed2c" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34621 } } peers { permanent_uuid: "44d1aa00c77946a8a0fbceed404bd1ec" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 39849 } } peers { permanent_uuid: "45e57f82ff1d42e0808bd13fb7d72e24" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 44249 } } }
I20260810 00:51:06.923607 31581 sys_catalog.cc:458] T 00000000000000000000000000000000 P adc0c8fd4d3e43a9abdf2a1df839ed2c [sys.catalog]: This master's current role is: LEADER
I20260810 00:51:06.930696 31417 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.6.60" port: 44249 } attempt: 5
W20260810 00:51:06.936439 31239 tablet.cc:2369] T 00000000000000000000000000000000 P adc0c8fd4d3e43a9abdf2a1df839ed2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 00:51:06 dist-test-slave-1nl6 krb5kdc[31178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786323066, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.6.61@KRBTEST.COM for kudu/127.28.6.60@KRBTEST.COM
I20260810 00:51:06.926232 31201 master_runner.cc:186] Error getting master registration for 127.28.6.61:39849: OK, instance_id { permanent_uuid: "44d1aa00c77946a8a0fbceed404bd1ec" instance_seqno: 1786323066208747 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 00:51:06.943701 31201 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 00:51:06.944134 31588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 00:51:06.946764 31588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 00:51:06.952268 31531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.932313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43818 (local address 127.28.6.60:44249)
0810 00:51:06.932549 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:06.932558 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:06.932582 (+ 24us) server_negotiation.cc:407] Connection header received
0810 00:51:06.932656 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:06.932664 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:06.932765 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:06.932900 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:06.934376 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:06.935958 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:06.938167 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:06.939212 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:06.939367 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:06.944624 (+ 5257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:51:06.944632 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:51:06.944637 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:51:06.944676 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:51:06.948575 (+ 3899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:06.949495 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:06.949502 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:06.949507 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:06.949669 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:06.950306 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:06.950324 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:06.950327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:06.950664 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:51:06.950783 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:06.951576 (+ 793us) server_negotiation.cc:299] Negotiation successful
0810 00:51:06.951807 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 00:51:06.953805 31419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.931411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.6.60:44249 (local address 127.0.0.1:43818)
0810 00:51:06.931859 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0810 00:51:06.931885 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 00:51:06.932022 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 00:51:06.933152 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 00:51:06.933160 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 00:51:06.933848 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0810 00:51:06.934138 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:51:06.934154 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:51:06.936172 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:51:06.936179 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:51:06.937938 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 00:51:06.937960 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 00:51:06.939628 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 00:51:06.939637 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 00:51:06.939918 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:06.940667 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 00:51:06.940698 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 00:51:06.944379 (+ 3681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 00:51:06.948789 (+ 4410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:51:06.948794 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:51:06.948799 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:51:06.949323 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:51:06.949853 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 00:51:06.949859 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 00:51:06.949863 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 00:51:06.950109 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 00:51:06.950945 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 00:51:06.950951 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 00:51:06.951214 (+ 263us) client_negotiation.cc:769] Sending connection context
0810 00:51:06.952824 (+ 1610us) client_negotiation.cc:240] Negotiation successful
0810 00:51:06.953236 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272}
I20260810 00:51:06.960239 31417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44d1aa00c77946a8a0fbceed404bd1ec: Bootstrap starting.
I20260810 00:51:06.964308 31417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44d1aa00c77946a8a0fbceed404bd1ec: Neither blocks nor log segments found. Creating new log.
I20260810 00:51:06.966590 31417 log.cc:824] T 00000000000000000000000000000000 P 44d1aa00c77946a8a0fbceed404bd1ec: Log is configured to *not* fsync() on all Append() calls
W20260810 00:51:06.971762 31457 tablet.cc:2369] T 00000000000000000000000000000000 P 45e57f82ff1d42e0808bd13fb7d72e24: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 00:51:06.977487 31489 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 45e57f82ff1d42e0808bd13fb7d72e24 [term 1 FOLLOWER]: Refusing update from remote peer adc0c8fd4d3e43a9abdf2a1df839ed2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:51:06.978529 31563 consensus_queue.cc:1035] T 00000000000000000000000000000000 P adc0c8fd4d3e43a9abdf2a1df839ed2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "45e57f82ff1d42e0808bd13fb7d72e24" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 44249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:51:06.990243 31542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.966014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52242 (local address 127.28.6.61:39849)
0810 00:51:06.966255 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:06.966263 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:06.969700 (+ 3437us) server_negotiation.cc:407] Connection header received
0810 00:51:06.970001 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:06.970009 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:06.970103 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:06.970185 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:06.976014 (+ 5829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:06.977042 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:06.978974 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:06.979682 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:06.979780 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:06.983207 (+ 3427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:51:06.983215 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:51:06.983221 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:51:06.983259 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:51:06.986942 (+ 3683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:06.987819 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:06.987825 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:06.987829 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:06.987976 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:06.988458 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:06.988463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:06.988467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:06.988809 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:51:06.989038 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:06.989659 (+ 621us) server_negotiation.cc:299] Negotiation successful
0810 00:51:06.989878 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260810 00:51:07.006313 31544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45e57f82ff1d42e0808bd13fb7d72e24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader adc0c8fd4d3e43a9abdf2a1df839ed2c. Latest consensus state: current_term: 1 leader_uuid: "adc0c8fd4d3e43a9abdf2a1df839ed2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc0c8fd4d3e43a9abdf2a1df839ed2c" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34621 } } peers { permanent_uuid: "44d1aa00c77946a8a0fbceed404bd1ec" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 39849 } } peers { permanent_uuid: "45e57f82ff1d42e0808bd13fb7d72e24" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 44249 } } }
I20260810 00:51:07.006659 31544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45e57f82ff1d42e0808bd13fb7d72e24 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 00:51:07.009049 31581 sys_catalog.cc:455] T 00000000000000000000000000000000 P adc0c8fd4d3e43a9abdf2a1df839ed2c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "adc0c8fd4d3e43a9abdf2a1df839ed2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc0c8fd4d3e43a9abdf2a1df839ed2c" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34621 } } peers { permanent_uuid: "44d1aa00c77946a8a0fbceed404bd1ec" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 39849 } } peers { permanent_uuid: "45e57f82ff1d42e0808bd13fb7d72e24" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 44249 } } }
I20260810 00:51:07.009339 31581 sys_catalog.cc:458] T 00000000000000000000000000000000 P adc0c8fd4d3e43a9abdf2a1df839ed2c [sys.catalog]: This master's current role is: LEADER
I20260810 00:51:07.009574 31591 mvcc.cc:205] Tried to move back new op lower bound from 7316779282329415680 to 7316779282024296448. Current Snapshot: MvccSnapshot[applied={T|T < 7316779282329415680}]
I20260810 00:51:07.010844 31599 mvcc.cc:205] Tried to move back new op lower bound from 7316779282329415680 to 7316779282024296448. Current Snapshot: MvccSnapshot[applied={T|T < 7316779282329415680}]
I20260810 00:51:07.013669 31417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44d1aa00c77946a8a0fbceed404bd1ec: No bootstrap required, opened a new log
I20260810 00:51:07.014247 31588 catalog_manager.cc:1347] Generated new cluster ID: 1e4945fd6e004ea9a990c7a6da34782c
I20260810 00:51:07.014362 31588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 00:51:07.028437 31544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45e57f82ff1d42e0808bd13fb7d72e24 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "adc0c8fd4d3e43a9abdf2a1df839ed2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc0c8fd4d3e43a9abdf2a1df839ed2c" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34621 } } peers { permanent_uuid: "44d1aa00c77946a8a0fbceed404bd1ec" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 39849 } } peers { permanent_uuid: "45e57f82ff1d42e0808bd13fb7d72e24" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 44249 } } }
I20260810 00:51:07.028723 31544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45e57f82ff1d42e0808bd13fb7d72e24 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 00:51:07.034107 31541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.959519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39040 (local address 127.28.6.62:34621)
0810 00:51:06.959719 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:06.959726 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:06.968272 (+ 8546us) server_negotiation.cc:407] Connection header received
0810 00:51:06.968964 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:06.968971 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:06.969073 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:06.969145 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:06.996646 (+ 27501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:06.997779 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:07.003396 (+ 5617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:07.004137 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:07.004281 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:07.024298 (+ 20017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:51:07.024306 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:51:07.024311 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:51:07.024355 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:51:07.028072 (+ 3717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:07.030495 (+ 2423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:07.030502 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:07.030506 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:07.030648 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:07.031745 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:07.031751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:07.031755 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:07.032089 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:51:07.032363 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:07.033438 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0810 00:51:07.033709 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 00:51:07.036638 31531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:06.968274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43834 (local address 127.28.6.60:44249)
0810 00:51:06.968505 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:06.968513 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:06.996819 (+ 28306us) server_negotiation.cc:407] Connection header received
0810 00:51:06.997658 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:06.997664 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:06.997748 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:06.997817 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:07.000051 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:07.001278 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:07.020568 (+ 19290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:07.021323 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:07.021427 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:07.029294 (+ 7867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 00:51:07.029301 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 00:51:07.029306 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 00:51:07.029352 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 00:51:07.033165 (+ 3813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:07.033980 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:07.033986 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:07.033990 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:07.034127 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 00:51:07.034579 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 00:51:07.034585 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 00:51:07.034589 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 00:51:07.035018 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 00:51:07.035264 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:07.035937 (+ 673us) server_negotiation.cc:299] Negotiation successful
0810 00:51:07.036221 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 00:51:07.044337 31417 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44d1aa00c77946a8a0fbceed404bd1ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc0c8fd4d3e43a9abdf2a1df839ed2c" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34621 } } peers { permanent_uuid: "44d1aa00c77946a8a0fbceed404bd1ec" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 39849 } } peers { permanent_uuid: "45e57f82ff1d42e0808bd13fb7d72e24" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 44249 } }
I20260810 00:51:07.044795 31417 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44d1aa00c77946a8a0fbceed404bd1ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d1aa00c77946a8a0fbceed404bd1ec, State: Initialized, Role: FOLLOWER
I20260810 00:51:07.045369 31417 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44d1aa00c77946a8a0fbceed404bd1ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc0c8fd4d3e43a9abdf2a1df839ed2c" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34621 } } peers { permanent_uuid: "44d1aa00c77946a8a0fbceed404bd1ec" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 39849 } } peers { permanent_uuid: "45e57f82ff1d42e0808bd13fb7d72e24" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 44249 } }
I20260810 00:51:07.049516 31606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44d1aa00c77946a8a0fbceed404bd1ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc0c8fd4d3e43a9abdf2a1df839ed2c" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34621 } } peers { permanent_uuid: "44d1aa00c77946a8a0fbceed404bd1ec" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 39849 } } peers { permanent_uuid: "45e57f82ff1d42e0808bd13fb7d72e24" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 44249 } } }
I20260810 00:51:07.050458 31606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44d1aa00c77946a8a0fbceed404bd1ec [sys.catalog]: This master's current role is: FOLLOWER
I20260810 00:51:07.050684 31417 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44d1aa00c77946a8a0fbceed404bd1ec [sys.catalog]: configured and running, proceeding with master startup.
I20260810 00:51:07.050761 31381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "adc0c8fd4d3e43a9abdf2a1df839ed2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "44d1aa00c77946a8a0fbceed404bd1ec"
I20260810 00:51:07.051442 31380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44d1aa00c77946a8a0fbceed404bd1ec [term 0 FOLLOWER]: Advancing to term 1
I20260810 00:51:07.056025 31588 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 00:51:07.057763 31588 catalog_manager.cc:1504] Loading token signing keys...
I20260810 00:51:07.058542 31380 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 44d1aa00c77946a8a0fbceed404bd1ec [term 1 FOLLOWER]: Refusing update from remote peer adc0c8fd4d3e43a9abdf2a1df839ed2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 00:51:07.059542 31581 consensus_queue.cc:1035] T 00000000000000000000000000000000 P adc0c8fd4d3e43a9abdf2a1df839ed2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d1aa00c77946a8a0fbceed404bd1ec" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 39849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 00:51:07.063639 31381 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 44d1aa00c77946a8a0fbceed404bd1ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adc0c8fd4d3e43a9abdf2a1df839ed2c in term 1.
I20260810 00:51:07.086982 31588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adc0c8fd4d3e43a9abdf2a1df839ed2c: Generated new TSK 0
I20260810 00:51:07.088090 31588 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 00:51:07.116505 31606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44d1aa00c77946a8a0fbceed404bd1ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader adc0c8fd4d3e43a9abdf2a1df839ed2c. Latest consensus state: current_term: 1 leader_uuid: "adc0c8fd4d3e43a9abdf2a1df839ed2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc0c8fd4d3e43a9abdf2a1df839ed2c" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34621 } } peers { permanent_uuid: "44d1aa00c77946a8a0fbceed404bd1ec" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 39849 } } peers { permanent_uuid: "45e57f82ff1d42e0808bd13fb7d72e24" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 44249 } } }
I20260810 00:51:07.116842 31606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44d1aa00c77946a8a0fbceed404bd1ec [sys.catalog]: This master's current role is: FOLLOWER
I20260810 00:51:07.117308 31616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44d1aa00c77946a8a0fbceed404bd1ec [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "adc0c8fd4d3e43a9abdf2a1df839ed2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc0c8fd4d3e43a9abdf2a1df839ed2c" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34621 } } peers { permanent_uuid: "44d1aa00c77946a8a0fbceed404bd1ec" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 39849 } } peers { permanent_uuid: "45e57f82ff1d42e0808bd13fb7d72e24" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 44249 } } }
I20260810 00:51:07.117694 31616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44d1aa00c77946a8a0fbceed404bd1ec [sys.catalog]: This master's current role is: FOLLOWER
I20260810 00:51:07.120357 31563 sys_catalog.cc:455] T 00000000000000000000000000000000 P adc0c8fd4d3e43a9abdf2a1df839ed2c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "adc0c8fd4d3e43a9abdf2a1df839ed2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc0c8fd4d3e43a9abdf2a1df839ed2c" member_type: VOTER last_known_addr { host: "127.28.6.62" port: 34621 } } peers { permanent_uuid: "44d1aa00c77946a8a0fbceed404bd1ec" member_type: VOTER last_known_addr { host: "127.28.6.61" port: 39849 } } peers { permanent_uuid: "45e57f82ff1d42e0808bd13fb7d72e24" member_type: VOTER last_known_addr { host: "127.28.6.60" port: 44249 } } }
I20260810 00:51:07.120666 31563 sys_catalog.cc:458] T 00000000000000000000000000000000 P adc0c8fd4d3e43a9abdf2a1df839ed2c [sys.catalog]: This master's current role is: LEADER
I20260810 00:51:07.153904 31629 catalog_manager.cc:1259] Loaded cluster ID: 1e4945fd6e004ea9a990c7a6da34782c
I20260810 00:51:07.154095 31629 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 44d1aa00c77946a8a0fbceed404bd1ec: loading cluster ID for follower catalog manager: success
I20260810 00:51:07.158262 31629 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 44d1aa00c77946a8a0fbceed404bd1ec: acquiring CA information for follower catalog manager: success
I20260810 00:51:07.161198 31629 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 44d1aa00c77946a8a0fbceed404bd1ec: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 00:51:07.259572 31541 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:51:07.231814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39054 (local address 127.28.6.62:34621)
0810 00:51:07.232028 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:07.232035 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:07.255417 (+ 23382us) server_negotiation.cc:407] Connection header received
0810 00:51:07.257007 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:07.257015 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:07.257112 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:07.257262 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:07.259269 (+ 2007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39054: BlockingRecv error: recv got EOF from 127.0.0.1:39054 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260810 00:51:07.261627 31542 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:51:07.238167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52248 (local address 127.28.6.61:39849)
0810 00:51:07.238397 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:07.238406 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:07.256374 (+ 17968us) server_negotiation.cc:407] Connection header received
0810 00:51:07.257542 (+ 1168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:07.257548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:07.257636 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:07.257711 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:07.261382 (+ 3671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52248: BlockingRecv error: recv got EOF from 127.0.0.1:52248 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260810 00:51:07.262380 31531 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:51:07.251372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43842 (local address 127.28.6.60:44249)
0810 00:51:07.251561 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:07.251567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:07.260388 (+ 8821us) server_negotiation.cc:407] Connection header received
0810 00:51:07.260678 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:07.260684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:07.260791 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:07.260875 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:07.262112 (+ 1237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43842: BlockingRecv error: recv got EOF from 127.0.0.1:43842 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 00:51:07.318616 31542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:07.298623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52256 (local address 127.28.6.61:39849)
0810 00:51:07.298868 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:07.298876 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:07.309699 (+ 10823us) server_negotiation.cc:407] Connection header received
0810 00:51:07.310265 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:07.310272 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:07.310495 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:07.310640 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:51:07.311491 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:07.312811 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:07.315170 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:07.315976 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:07.316079 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:07.316924 (+ 845us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:51:07.317690 (+ 766us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:51:07.317851 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:07.318113 (+ 262us) server_negotiation.cc:299] Negotiation successful
0810 00:51:07.318210 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
W20260810 00:51:07.361860 31531 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:51:07.352333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43846 (local address 127.28.6.60:44249)
0810 00:51:07.352599 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:07.352606 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:07.359543 (+ 6937us) server_negotiation.cc:407] Connection header received
0810 00:51:07.360119 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:07.360126 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:07.360287 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:07.360369 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:07.361563 (+ 1194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43846: BlockingRecv error: recv got EOF from 127.0.0.1:43846 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 00:51:07.415839 31542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:07.396546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47480 (local address 127.28.6.61:39849)
0810 00:51:07.396765 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:07.396772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:07.407686 (+ 10914us) server_negotiation.cc:407] Connection header received
0810 00:51:07.408269 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:07.408277 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:07.408388 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:07.408467 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:51:07.409308 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:07.410629 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:07.413080 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:07.413943 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:07.414081 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:07.414833 (+ 752us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:51:07.415031 (+ 198us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:51:07.415107 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:07.415383 (+ 276us) server_negotiation.cc:299] Negotiation successful
0810 00:51:07.415489 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260810 00:51:07.465380 31531 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:51:07.455897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48310 (local address 127.28.6.60:44249)
0810 00:51:07.456120 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:07.456127 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:07.463053 (+ 6926us) server_negotiation.cc:407] Connection header received
0810 00:51:07.463621 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:07.463629 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:07.463760 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:07.463856 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:07.465109 (+ 1253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48310: BlockingRecv error: recv got EOF from 127.0.0.1:48310 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 00:51:07.480166 31542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:07.472503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47492 (local address 127.28.6.61:39849)
0810 00:51:07.472691 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:07.472698 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:07.473219 (+ 521us) server_negotiation.cc:407] Connection header received
0810 00:51:07.473538 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:07.473544 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:07.473651 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:07.473756 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:51:07.474334 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:07.475494 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:07.477764 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:07.478440 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:07.478585 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:07.479217 (+ 632us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:51:07.479398 (+ 181us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:51:07.479528 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:07.479702 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 00:51:07.479810 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260810 00:51:07.490232 31531 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:51:07.487438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48312 (local address 127.28.6.60:44249)
0810 00:51:07.487633 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:07.487640 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:07.488272 (+ 632us) server_negotiation.cc:407] Connection header received
0810 00:51:07.488606 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:07.488613 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:07.488731 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:07.488825 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:07.489955 (+ 1130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48312: BlockingRecv error: recv got EOF from 127.0.0.1:48312 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 00:51:07.506634 31542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:07.498503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47508 (local address 127.28.6.61:39849)
0810 00:51:07.498797 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:07.498805 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:07.499469 (+ 664us) server_negotiation.cc:407] Connection header received
0810 00:51:07.499693 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:07.499698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:07.499784 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:07.499887 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:51:07.500398 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:07.501651 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:07.503987 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:07.504681 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:07.504840 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:07.505456 (+ 616us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:51:07.505800 (+ 344us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:51:07.505933 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:07.506218 (+ 285us) server_negotiation.cc:299] Negotiation successful
0810 00:51:07.506311 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
W20260810 00:51:07.519482 31531 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:51:07.516666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48318 (local address 127.28.6.60:44249)
0810 00:51:07.516971 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:07.516978 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:07.517542 (+ 564us) server_negotiation.cc:407] Connection header received
0810 00:51:07.517890 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:07.517898 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:07.518023 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:07.518109 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:07.519256 (+ 1147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48318: BlockingRecv error: recv got EOF from 127.0.0.1:48318 (error 108)
Metrics: {"server-negotiator.queue_time_us":180}
I20260810 00:51:07.539598 31542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:07.532001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47516 (local address 127.28.6.61:39849)
0810 00:51:07.532192 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:07.532198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:07.532773 (+ 575us) server_negotiation.cc:407] Connection header received
0810 00:51:07.532958 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:07.532966 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:07.533111 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:07.533241 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:51:07.533949 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:07.535290 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:07.537307 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:07.538065 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:07.538215 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:07.538707 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:51:07.538868 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:51:07.539002 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:07.539176 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 00:51:07.539268 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260810 00:51:07.555619 31531 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:51:07.552978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48332 (local address 127.28.6.60:44249)
0810 00:51:07.553179 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:07.553184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:07.553784 (+ 600us) server_negotiation.cc:407] Connection header received
0810 00:51:07.554074 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:07.554080 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:07.554191 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:07.554270 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:07.555363 (+ 1093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48332: BlockingRecv error: recv got EOF from 127.0.0.1:48332 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 00:51:07.584376 31542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:07.576789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47530 (local address 127.28.6.61:39849)
0810 00:51:07.577040 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:07.577046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:07.577648 (+ 602us) server_negotiation.cc:407] Connection header received
0810 00:51:07.577908 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:07.577915 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:07.578008 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:07.578132 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:51:07.578695 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:07.579901 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:07.581923 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:07.582645 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:07.582776 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:07.583429 (+ 653us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:51:07.583576 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:51:07.583682 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:07.583854 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 00:51:07.584000 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260810 00:51:07.602162 31531 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:51:07.599404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48348 (local address 127.28.6.60:44249)
0810 00:51:07.599600 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:07.599607 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:07.600202 (+ 595us) server_negotiation.cc:407] Connection header received
0810 00:51:07.600529 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:07.600536 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:07.600672 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:07.600759 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:07.601896 (+ 1137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48348: BlockingRecv error: recv got EOF from 127.0.0.1:48348 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 00:51:07.645722 31542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:07.638585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47540 (local address 127.28.6.61:39849)
0810 00:51:07.638791 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:07.638808 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:07.639925 (+ 1117us) server_negotiation.cc:407] Connection header received
0810 00:51:07.640589 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:07.640595 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:07.640685 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:07.640800 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:51:07.641182 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:07.642388 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:07.643822 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:07.644487 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:07.644630 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:07.644972 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:51:07.645115 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:51:07.645215 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:07.645356 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 00:51:07.645446 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260810 00:51:07.653749 31531 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:51:07.651781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48356 (local address 127.28.6.60:44249)
0810 00:51:07.651991 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:07.651996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:07.652336 (+ 340us) server_negotiation.cc:407] Connection header received
0810 00:51:07.652500 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:07.652504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:07.652576 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:07.652659 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:07.653505 (+ 846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48356: BlockingRecv error: recv got EOF from 127.0.0.1:48356 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260810 00:51:07.726227 31542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:07.721283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47554 (local address 127.28.6.61:39849)
0810 00:51:07.721543 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:07.721550 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:07.721819 (+ 269us) server_negotiation.cc:407] Connection header received
0810 00:51:07.721980 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:07.721984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:07.722052 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:07.722132 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:51:07.722541 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:07.723333 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:07.724650 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:07.725062 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:07.725159 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:07.725514 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:51:07.725619 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:51:07.725683 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:07.725823 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 00:51:07.725885 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260810 00:51:07.733536 31531 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 00:51:07.731797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48360 (local address 127.28.6.60:44249)
0810 00:51:07.731961 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:07.731966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:07.732372 (+ 406us) server_negotiation.cc:407] Connection header received
0810 00:51:07.732521 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:07.732524 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:07.732608 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:07.732692 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 00:51:07.733370 (+ 678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48360: BlockingRecv error: recv got EOF from 127.0.0.1:48360 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 00:51:07.838042 31559 catalog_manager.cc:1259] Loaded cluster ID: 1e4945fd6e004ea9a990c7a6da34782c
I20260810 00:51:07.838160 31559 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 45e57f82ff1d42e0808bd13fb7d72e24: loading cluster ID for follower catalog manager: success
I20260810 00:51:07.840838 31559 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 45e57f82ff1d42e0808bd13fb7d72e24: acquiring CA information for follower catalog manager: success
I20260810 00:51:07.842618 31559 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 45e57f82ff1d42e0808bd13fb7d72e24: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 00:51:07.871016 31542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:07.865459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47562 (local address 127.28.6.61:39849)
0810 00:51:07.865650 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:07.865655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:07.866159 (+ 504us) server_negotiation.cc:407] Connection header received
0810 00:51:07.866486 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:07.866490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:07.866563 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:07.866654 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:51:07.867135 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:07.867870 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:07.869363 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:07.869803 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:07.869896 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:07.870343 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:51:07.870445 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:51:07.870512 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:07.870660 (+ 148us) server_negotiation.cc:299] Negotiation successful
0810 00:51:07.870725 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 00:51:07.883411 31531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:07.877224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48362 (local address 127.28.6.60:44249)
0810 00:51:07.877478 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:07.877485 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:07.877764 (+ 279us) server_negotiation.cc:407] Connection header received
0810 00:51:07.877989 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:07.877995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:07.878191 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:07.878332 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:51:07.878761 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:07.879558 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:07.880854 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:07.881314 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:07.881450 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:07.881999 (+ 549us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:51:07.882662 (+ 663us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:51:07.882849 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:07.882910 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 00:51:07.883027 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260810 00:51:07.897846 31542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:07.889825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47566 (local address 127.28.6.61:39849)
0810 00:51:07.890233 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:07.890241 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:07.891417 (+ 1176us) server_negotiation.cc:407] Connection header received
0810 00:51:07.891627 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:07.891636 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:07.891768 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:07.891874 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:51:07.892645 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:07.893642 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:07.895642 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:07.896296 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:07.896439 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:07.896959 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:51:07.897095 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:51:07.897176 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:07.897409 (+ 233us) server_negotiation.cc:299] Negotiation successful
0810 00:51:07.897498 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261}
I20260810 00:51:07.898490 31531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:07.890019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48368 (local address 127.28.6.60:44249)
0810 00:51:07.890411 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:07.890419 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:07.892147 (+ 1728us) server_negotiation.cc:407] Connection header received
0810 00:51:07.892367 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:07.892373 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:07.892481 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:07.892576 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:51:07.893284 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:07.894366 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:07.896311 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:07.896997 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:07.897132 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:07.897705 (+ 573us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:51:07.897851 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:51:07.897921 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:07.898167 (+ 246us) server_negotiation.cc:299] Negotiation successful
0810 00:51:07.898248 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245}
I20260810 00:51:07.899945 31787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 00:51:07.890196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52004 (local address 127.28.6.62:34621)
0810 00:51:07.891761 (+ 1565us) server_negotiation.cc:206] Beginning negotiation
0810 00:51:07.891767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 00:51:07.891787 (+ 20us) server_negotiation.cc:407] Connection header received
0810 00:51:07.891922 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 00:51:07.891927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 00:51:07.892190 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 00:51:07.892404 (+ 214us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 00:51:07.893385 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:07.894491 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:07.896567 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 00:51:07.897052 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 00:51:07.897180 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 00:51:07.897917 (+ 737us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 00:51:07.898486 (+ 569us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 00:51:07.898632 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 00:51:07.898978 (+ 346us) server_negotiation.cc:299] Negotiation successful
0810 00:51:07.899066 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1411,"thread_start_us":1298,"threads_started":1}
I20260810 00:51:07.904760 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 31201
I20260810 00:51:07.905858 31299 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:51:08.018589 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 31201
I20260810 00:51:08.050546 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 31319
I20260810 00:51:08.051819 31411 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:51:08.165181 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 31319
I20260810 00:51:08.193735 28696 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksi0adg/build/asan/bin/kudu pid 31427
I20260810 00:51:08.194988 31518 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 00:51:08.302367 28696 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksi0adg/build/asan/bin/kudu with pid 31427
2026-08-10T00:51:08Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4651 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9479 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (381047 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST